Data science tools

Engineering productivity tips

Star

terminal command prompt bash tmux git vim jupyter notebook

By Afshine Amidi and Shervine Amidi

Working in groups with Git

Overview Git is a version control system (VCS) that tracks changes of different files in a given repository. In particular, it is useful for:

• keeping track of file versions
• working in parallel thanks to the concept of branches
• backing up files to a remote server

Getting started The table below summarizes the commands to start a new project, depending on whether or not the repository already exists:

Case	Action	Command	Illustration
No existing repository	Initialize repository from local folder	git init
Repository already exists	Copy repository from remote to local	git clone path/to/address.git

File check-in We can track modifications made in the repository, done by either modifying, adding or deleting a file, through the following steps:

Step	Command	Illustration
1. Add modified, new, or deleted file to staging area	git add file
2. Save snapshot along with descriptive message	git commit -m 'description'

Remark 1: git add . will have all modified files to the staging area.

Remark 2: files that we do not want to track can be listed in the .gitignore file.

Sync with remote The following commands enable changes to be synchronized between remote and local machines:

Action	Command	Illustration
Fetch most recent changes from remote branch	git pull origin name_of_branch
Push latest local changes to remote branch	git push origin name_of_branch

Parallel workstreams In order to make changes that do not interfere with the current branch, we can create another branch name_of_new_branch as follows:

git checkout -b name_of_new_branch   # Create and checkout to that branch

Action	Command	Illustration
Merge name_of_branch with current branch	git merge name_of_branch
Remove name_of_branch	git branch -D name_of_branch

Tracking status We can check previous changes made to the repository with the following commands:

Action	Command	Illustration
Check status of modified file(s)	git status
View last commits	git log --oneline
Compare changes made between two commits	git diff commit_1 commit_2
View list of local branches	git branch

Canceling changes Canceling changes is done differently depending on the situation that we are in. The table below sums up the most common cases:

Case	Action	Command	Illustration
Unstaged	Revert file to state in last commit	git checkout -- file
Staged	Remove file from staging area	git reset HEAD file
Committed	Go back to a previous commit	git reset --hard prev_commit

Project structure It is important to keep a consistent and logical structure of the project. One example of structure of a data science project is as follows:

my_project/
  ├── analysis/
    ├── graphs/
    └── notebooks/
  ├── data/
    ├── query/
    ├── raw/
    └── processed/
  ├── modeling/
    ├── methods/
    ├── results/
    └── tests/
  └── README.md

Working with Bash

Basic terminal commands The table below sums up the most useful terminal commands:

Chaining It is a concept that improves readability by chaining operations with the pipe | operator. A few common examples are summed up in the table below:

Advanced search The find command allows the search of specific files and manipulate them if necessary. The general structure of the command is as follows:

find path_to_folder/. [conditions] [actions]

The possible conditions and actions are summarized in the table below:

Remark: the flags above can be combined to make a multi-condition search.

Changing permissions The following command enables to change the permissions of a given file (or folder):

chmod (-R) three_digits file

• the first digit is about the owner associated to the file
• the second digit is about the group associated to the file
• the third digit is anyone irrespective of their relation to the file

Each digit is one of (0, 4, 5, 6, 7), and has the following meaning:

For instance, giving read, write, execution permissions to everyone for a given_file is done by running the following command:

chmod 777 given_file

Remark: in order to change ownership of a file to a given user and group, we use the command chown user:group file.

Terminal shortcuts The table below summarizes the main shortcuts when working with the terminal:

Automating tasks

Create aliases Shortcuts can be added to the ~/.bash_profile file by adding the following line of code:

shortcut="command"

Bash scripts Bash scripts are files whose file name ends with .sh and where the file itself is structured as follows:

#!/bin/bash
... [bash script] ...

Crontabs By letting the day of the month vary between 1-31 and the day of the week vary between 0-6 (Sunday-Saturday), a crontab is of the following format:

  *         *         *         *         *
minute    hour       day      month      day
                   of month            of week

tmux Terminal multiplexing, often known as tmux, is a way of running tasks in the background and in parallel. The table below summarizes the main commands:

Mastering editors

Vim Vim is a popular terminal editor enabling quick and easy file editing, which is particularly useful when connected to a server. The main commands to have in mind are summarized in the table below:

Jupyter notebook Editing code in an interactive way is easily done through jupyter notebooks. The main commands to have in mind are summarized in the table below: