Photon



Photon can extract the following data while crawling:

URLs (in-scope & out-of-scope)
URLs with parameters (example.com/gallery.php?id=2)
Intel (emails, social media accounts, amazon buckets etc.)
Files (pdf, png, xml etc.)
Secret keys (auth/API keys & hashes)
JavaScript files & Endpoints present in them
Strings matching custom regex pattern
Subdomains & DNS related data
The extracted information is saved in an organized manner or can be exported as json.

Python Regular Expression

A Regular Expression, is a sequence of characters that forms a search pattern. Python has a built-in package called re, which can be used to work with Regular Expressions.

(Example Code to remove symbols and numbers - compatible with Python 2.7.17)
import re
import string
input_str = "58597884|01:31:50|The rise of python stated by pythonforengineers blog"
print"Before Processing:",input_str
result = re.sub(r'\d+', '', input_str)
result = result.translate(string.maketrans("",""), string.punctuation)
print"After Processing:",result

Python Open Source Project - Manim


Manim is an animation engine for explanatory math videos. It's used to create precise animations programmatically. 

Install it from PyPI via pip:        pip3 install manimlib

Click here for more info

Surprise - A Python scikit for recommender systems

Surprise is a Python scikit building and analyzing recommender systems that deal with explicit rating data. Surprise stands for Simple Python Recommendation System Engine.

Surprise was designed with the following purposes in mind:

Give users perfect control over their experiments.
Alleviate the pain of dataset handling.
Provide various ready-to-use prediction algorithms.
Make it easy to implement new algorithm ideas.
Provide tools to evaluate, analyse and compare the algorithms performance.

For more info http://surpriselib.com

Performing Google Search using Python code

Install following two packages before executing the code below pip install beautifulsoup4 pip install google try: from googlesearch import search except ImportError: print("No module named 'google' found") # to search query = "python for engineers Blog" for j in search(query, tld="co.in", num=10, stop=1, pause=2): print(j)

Pycharm - The Python IDE for Professional Developers

Pycharm - Features

Intelligent Python Assistance
PyCharm provides smart code completion, code inspections, on-the-fly error highlighting and quick-fixes, along with automated code refactorings and rich navigation capabilities.

Web Development Frameworks
PyCharm offers great framework-specific support for modern web development frameworks such as Django, Flask, Google App Engine, Pyramid, and web2py.

Scientific Tools
PyCharm integrates with IPython Notebook, has an interactive Python console, and supports Anaconda as well as multiple scientific packages including matplotlib and NumPy.

Cross-technology Development
In addition to Python, PyCharm supports JavaScript, CoffeeScript, TypeScript, Cython, SQL, HTML/CSS, template languages, AngularJS, Node.js, and more.

Remote Development Capabilities
Run, debug, test, and deploy applications on remote hosts or virtual machines, with remote interpreters, an integrated ssh terminal, and Docker and Vagrant integration.

Built-in Developer Tools
A huge collection of tools out of the box: an integrated debugger and test runner; Python profiler; a built-in terminal; and integration with major VCS and built-in Database Tools.


Pylint - Python source code analyser

Pylint is a Python source code analyser which looks for programming errors, helps to enforce a coding standard, and other such. This quality checker for Python programming includes several features such as coding standard where it checks for the length of line codes, error detection, refactoring by detecting the duplicated code, Pylint is shipped with Pyreverse which creates UML diagrams for python code.

Zerynth enables Python for Microcontrollers & middleware for IoT and Industry 4.0


Zerynth speeds up IoT and embedded development to the huge community of Python programmers. With Zerynth you can program in Python or hybrid C/Python language the most popular 32-bit microcontrollers, and connect them to the top Cloud infrastructures.

Zerynth allows a mixed C/Python programming environment. This kind of “hybrid” programming is extremely powerful in scenarios where the project needs low-level code for time-critical tasks while keeping Python flexibility and readability for not time-critical sections.  For more info

Walrus Operator in Python 3.8

One of the biggest highlights of Python 3.8.0 is a new feature for assignment expressions known as the Walrus Operator.

"There is new syntax := that assigns values to variables as part of a larger expression," the Python 3.8.0 release notes state.


"It is affectionately known as 'the walrus operator' due to its resemblance to the eyes and tusks of a walrus."



In this example, the assignment expression helps avoid calling len() twice:

if (n := len(a)) > 10:
    print(f"List is too long ({n} elements, expected <= 10)")

Python interface to MATLAB (pymatlab)

This package lets Python users interface and communicate with MATLAB from Python. Pymatlab makes it easier for users to integrate a project with a large MATLAB codebase into python scripts by using MATLAB scripts as a part of the python program.

Is Einstein Analytics shifting from hybrid C-Python towards Google's Go language ?

According to Salesforce principal architect Guillaume Le Stum, before Salesforce launched Einstein Analytics, the query engine and dataset creation tools that eventually became Einstein Analytics were written in C "for performance" and a Python wrapper that provided functionality like parsing queries, and a REST API server. 
But before launch, Le Stum says the guts of Einstein Analytics started to show performance slowdowns because any new feature that wasn't part of the core query engine got loaded onto the Python wrapper and shifted to the Go application. 
Is the Le Stum claim of "Python doesn't do multi-threading very well, so the more the wrapper was being asked to do, the worse it performed," is acceptable. 
News 

Microsoft: We want you to learn Python programming language for free

Microsoft has launched a new 44-part series called Python for Beginners on YouTube, consisting of three- to four-minute lessons from two self-described geeks at Microsoft who love programming and teaching in YouTube or  Channel9

The Python for Beginners series is presented by Christopher Harrison, a senior program manager at Microsoft, and Susan Ibach, a business development manager from Microsoft's AI Gaming unit

Microsoft has published a page on GitHub containing additional resources, including slides and code samples to help students become better at Python. 

The Developers have a option to interface with machine-learning frameworks like Google-developed TensorFlow, and the Microsoft Cognitive Toolkit (CNTK)




Python in top place in the newest annual ranking of popular programming languages by IEEE Spectrum

Programming language popularity: Python tightens its grip at the top. Python retains its top spot as the most popular language for electrical and electronics engineers.

Interactive: The Top Programming Languages

DDA Line Drawing Algorithms Line Coordinates

def ROUND(a):
  return int(a + 0.5)
def drawDDA(x1,y1,x2,y2):
  x,y = x1,y1
  length = (x2-x1) if (x2-x1) > (y2-y1) else (y2-y1)
  dx = (x2-x1)/float(length)
  dy = (y2-y1)/float(length)
  print ('x = %s, y = %s' % (((ROUND(x),ROUND(y)))))
  for i in range(length):
    x += dx
    y += dy
    print ('x = %s, y = %s' % (((ROUND(x),ROUND(y)))))
drawDDA(2,5,10,20)

PyRobot - Python for Robotics

PyRobot is a Python package for benchmarking and running experiments in robot learning. The goal of this project is to abstract away the low-level controls for individual robots from the high-level motion generation and learning in an easy-to-use way. Using PyRobot will allow you to run robots without having to deal with the robot specific software along with enabling better comparisons.




Building Standalone Python Applications with PyOxidizer

PyOxidizer is a utility for producing binaries that embed Python. The over-arching goal of PyOxidizer is to make complex packaging and distribution problems simple so application maintainers can focus on building applications instead of toiling with build systems and packaging tools.

PyOxidizer is capable of producing a single file executable - with a copy of Python and all its dependencies statically linked and all resources (like .pyc files) embedded in the executable. You can copy a single executable file to another machine and run a Python application contained within. It just works.

Msticpy - A Python Defender Tool For Security Investigations & Hunting

Microsoft Threat Intelligence Python Security Tools

The msticpy package was initially developed to support Jupyter Notebook authoring for Azure Sentinel. Many of the included tools can be used in other security scenarios for threat hunting and threat investigation. There are three main sub-packages:

  • sectools - python security tools to help with data analysis or investigation
  • nbtools - Jupyter-specific UI tools such as widgets and data display
  • data - data interfaces specific to Sentinel/Log Analytics

Installing

pip install msticpy
pip install git+https://github.com/microsoft/msticpy

Python continues to soar in the TIOBE index



In June 2019, Python has reached again an all time high in TIOBE index of 8.5%. If Python can keep this pace, it will probably replace C and Java in 3 to 4 years time, thus becoming the most popular programming language of the world. The main reason for this is that software engineering is booming. It attracts lots of newcomers to the field. 

More Details 

Python developers reveal their favorite tool kits

  • Python is used mainly for Data Analysis 
  • NumPy is most popular data science framework 
  • Flask is most popular web frameworks 
  • Requests is most popular software libraries 
  • PyCharm is most popular IDEs for Python 

Autocomplete Coding Tools For Python Programmers

kite - Code Faster in Python with Line-of-Code Completions
Jedi - An awesome auto completion/static analysis library for Python
Wing - The Intelligent Development Environment for Python
Finisher - It is a lightweight autocompletion library for Python.

Python program to print memory and processor usage

import os import psutil pid = os.getpid() py = psutil.Process(pid) mu = (py.memory_info()[0] / 2.**30) * 1000 print('Memory Use(MB):', mu, 'of process id:', pid) print('CPU Use:',psutil.cpu_percent())

Python at Netflix

Netflix relies on Python as programming language gains industry prominence

Netflix relies heavily on Python, using the programming language for its ​"full content lifecycle,​" including tasks like security automation and training machine learning models for its recommendation algorithms, according to a Netflix Technology Blog Tuesday.

News @ ciodrive

News @ Netflix Tech Blog 

Python Program to Find the Fibonacci Series with defined program storage infinity

a=0 b=1 n=9999999999999999999999 # Number of terms in Fibonacci Series (sample value) # An integer giving the maximum value a variable of type Py_ssize_t can take. # It 's usually 2^31 - 1 on a 32-bit platform and 2^63 - 1 on a 64-bit platform. print(a,b,end=" ") while(n-2): c=a+b a=b b=c print("\n",c) n=n-1

Top 40 Python Blogs, Websites And Newsletters To Follow in 2019

Python Blogs List The Best Python blogs from thousands of top Python blogs in our index using search and social metrics. The python for engineers is listed at 17th position. 

Python Web Development Frameworks

Full Stack Frameworks

It gives full support to developers including basic components like form generators, form validation, and template layouts etc

1. Django is a high-level Python Web application development framework that encourages us to develop things rapidly, It uses pragmatic design.  It’s free and open source.

2. Web2py is a free open source full-stack development framework in python which allows the user to develop things quickly. It is a cross-platform framework that supports all popular operating systems. 

3. TurboGears is a free, open source and data-driven full-stack web application development Python framework. With the help of Javascript developer tools, developers can simply the web application.

4. CubicWeb is a semantic, free and open-source Python web framework, that empowers developers to efficiently build web applications by reusing components and following the well known object-oriented design principles. 

Non Full Stack Frameworks

Non-full stack frameworks are also called as Micro frameworks because it doesn’t have many components like full stack frameworks.

1. Flask is a microframework for Python based on Werkzeug, and Jinja 2. The main purpose is to develop a strong web application base. As compared to Django, Flask is best suited for small and easy projects.

2. CherryPy is a Minimalist Python Web Framework. It uses the Object-Oriented paradigm to develop web applications. This approach helps developers to develop web applications within a short period of time.

3. Bottle is a fast, simple and lightweight WSGI micro web-framework for Python. It is distributed as a single file module and has no dependencies other than the Python Standard Library. It is an easy-to-use lightweight framework generally used to build small web applications. It is mainly used to develop API’s.

4. Tornado is a python web framework with asynchronous network library. By using the non-blocking network I/O, Tornado can scale to tens of thousands of open connections, making it ideal for long polling, WebSockets, and other applications that require a long-lived connection to each user.

Top 20 Python libraries

1. Requests. The most famous http library written by kenneth reitz.

2. Scrapy. If you are involved in webscraping then this is a must have library for you.

3. wxPython. A gui toolkit for python. I have primarily used it in place of tkinter.

4. Pillow. A friendly fork of PIL (Python Imaging Library).

5. SQLAlchemy. A database library.

6. BeautifulSoup. The xml and html parsing library is very useful for beginners.

7. Twisted. The most important tool for any network application developer.

8. NumPy.  It provides some advance math functionalities to python.

9. SciPy.  It is a library of algorithms and mathematical tools for python.

10. matplotlib. A numerical plotting library. It is very useful for any data scientist or any data analyzer.

11. Pygame.  This library will help you achieve your goal of 2d game development.

12. Pyglet. A 3d animation and game creation engine.

13. pyQT. A GUI toolkit for python.

14. pyGtk.  It is the same library in which the famous Bittorrent client is created.

15. Scapy. A packet sniffer and analyzer for python made in python.

16. pywin32. A python library which provides some useful methods and classes for interacting with windows.

17. nltk. Natural Language Toolkit

18. nose. A testing framework for python.

19. SymPy. SymPy can do algebraic evaluation, differentiation, expansion, complex numbers, etc.

20. IPython. It has completion, history, shell capabilities, and a lot more.

Artificial Intelligence and Python in school curriculum to make students future-ready

Familiarizing students with new technologies will make the teaching learning process fruitful. The students with all background can write program to stimulate, what they have studied. It should be a mutual learning like both python and the subject they have implemented.  More

Python inEducation - Teach, Learn, Program 

A book on Python in Education by O’Reilly authored by Nicholas H. Tollervey

Python program to renaming the file or directory

# X is the original name and Y is the name to be changed
import os, sys
print ("The dir is: %s"%os.listdir(os.getcwd()))
os.rename("X","Y")  
print ("Successfully renamed.")
print ("the dir is: %s" %os.listdir(os.getcwd()))

Python Libraries for General AI

  • AIMA – Python implementation of algorithms from Russell and Norvig’s ‘Artificial Intelligence: A Modern Approach’
  • pyDatalog – Logic Programming engine in Python
  • SimpleAI – Python implementation of many of the artificial intelligence algorithms described on the book “Artificial Intelligence, a Modern Approach”. It focuses on providing an easy to use, well documented and tested library.
  • EasyAI – Simple Python engine for two-players games with AI (Negamax, transposition tables, game solving).