Skip to content
Snippets Groups Projects
Commit 7ebd7487 authored by Max Melchert's avatar Max Melchert
Browse files

feat: allow specifying individual files for transfer to server

parent fcd9f807
No related branches found
No related tags found
No related merge requests found
......@@ -45,7 +45,7 @@ Vorteile von _Byobu_:
1. Makefile [herunterladen](./basic.mk)
2. Makefile in Projektordner speichern
- Projektstruktur ([Beispiel](#projektstruktur))
3. Makefile in `Makefile` oder `makefile` umbenennen
3. Makefile in `Makefile` oder `makefile` umbenennen (ohne `.mk`)
4. Makefile [konfigurieren](#konfiguration)
5. `make install`
......@@ -122,21 +122,49 @@ Sie werden durch den Aufruf von `make download` im zuvor definierten Ordner `LOC
## Konfiguration
### Server
| Variable | Verwendung | Beispiel |
| :-------------------- | :--------------------------------- | :----------------------------------- |
| :------------ | :---------------- | :----------------- |
| `SERVER_USER` | Server: User Name | `mmustermann` |
| `SERVER_HOST` | Server: Host Name | `hyrican-1-extern` |
| `CONDA_PATH` | Server: Conda-Installations-Pfad | `/home/mmustermann/.local/opt/conda` |
| `CONDA_ENV` | Server: Conda-Environment-Name | `ml` |
### Python Environment
| Variable | Verwendung | Beispiel |
| :---------------- | :------------------------------- | :--------------------------------- |
| `CONDA_PATH` | Server: Conda-Installations-Pfad | `/opt/conda` |
| `CONDA_ENV` | Server: Conda-Environment-Pfad | `~/envs/ml` |
| `PYTHON_VERSION` | Python: Gewünschte Version | `3.10` |
| `PYTHON_PACKAGES` | Python: Benötigte Pakete | `tensorflow-cpu matplotlib optuna` |
### Project Paths
| Variable | Verwendung | Beispiel |
| :------------------- | :------------------------- | :------------------- |
| `LOCAL_PROJECT_DIR` | Client: Projektordner [^1] | `.` |
| `REMOTE_PROJECT_DIR` | Server: Projektordner [^2] | `~s/akis/my_project` |
### Project Files
| Variable | Verwendung | Beispiel |
| :------------ | :-------------------------------- | :------------- |
| `MAIN_SCRIPT` | Auszuführendes Python-Script [^1] | `main.py` |
| `LOG_FILE` | Log-Datei (optional) [^1] | `progress.log` |
| `LOCAL_PROJECT_DIR` | Client: Projektordner [^1] | `.` |
| `LOCAL_OUTPUT_DIR` | Client: Output-Ordner [^1] | `output` |
| `REMOTE_PROJECT_DIR` | Server: Projektordner [^2] | `/home/mmustermann/akis/my_project` |
| `REMOTE_OUTPUT_FILES` | Server: Output-Dateien [^1] | `output/*.svg output/*pkl` |
| `COPY_LINKS_AS_FILES` | Server: Kopiert Datei des Symlinks | `false` |
### File Transfer
| Variable | Verwendung | Beispiel |
| :---------------------- | :---------------------------- | :--------------------------- |
| `LOCAL_UPLOAD_FILES` | Client: Upload-Dateien [^1] | `main.py scripts data/*.pkl` |
| `LOCAL_DOWNLOAD_DIR` | Client: Download-Ordner [^1] | `output` |
| `REMOTE_DOWNLOAD_FILES` | Server: Download-Dateien [^1] | `output/*.svg output/*.pkl` |
### Misc
| Variable | Verwendung | Beispiel |
| :-------------------- | :--------------------------------- | :------- |
| `COPY_LINKS_AS_FILES` | Server: Kopiert Symlinks als Datei | `false` |
[^1]: Relativer Pfad zum Projektordner
[^2]: Absoluter Pfad
......@@ -165,6 +193,7 @@ Sie werden durch den Aufruf von `make download` im zuvor definierten Ordner `LOC
- [ ] Slack
- [ ] Matrix
- [ ] parallele Ausführung mehrerer Scripts
- [ ] docs: get local python packages; append to `PYTHON_PACKAGES`
---
......
......@@ -13,19 +13,20 @@ PYTHON_PACKAGES := tensorflow-cpu ipympl mypy jupyter pystan \
geopandas descartes pyogrio optuna plotly tqdm spyder \
jax-jumpy moviepy shimmy-atari gymnasium-box2d \
ipywidgets notebook lxml fastparquet zstandard \
contextily networkx pyvis netgraph pip
contextily networkx pyvis netgraph keras pip
## Project Paths
LOCAL_PROJECT_DIR := .
REMOTE_PROJECT_DIR := ~/akis/my_project
## Project Files
MAIN_SCRIPT := main.py
LOG_FILE := progress.log
## Local Project Paths
LOCAL_PROJECT_DIR := .
LOCAL_OUTPUT_DIR := output
## Remote Project Paths
REMOTE_PROJECT_DIR := ~/my_project
REMOTE_OUTPUT_FILES := output/*
## File Transfer
LOCAL_UPLOAD_FILES := main.py scripts data/*.pkl
LOCAL_DOWNLOAD_DIR := output
REMOTE_DOWNLOAD_FILES := output/*.svg output/*.pkl
## Misc
COPY_LINKS_AS_FILES := false
......@@ -34,7 +35,7 @@ COPY_LINKS_AS_FILES := false
LOGIN := $(SERVER_USER)@$(SERVER_HOST)
SSH := ssh $(LOGIN)
RSYNC := rsync --archive --compress --progress $(if $(filter true,$(COPY_LINKS_AS_FILES)),--copy-links)
RSYNC := rsync --relative --archive --compress --progress $(if $(filter true,$(COPY_LINKS_AS_FILES)),--copy-links)
MAMBA := source $(CONDA_PATH)/etc/profile.d/conda.sh && mamba # TODO: mamba.sh
CONDA := source $(CONDA_PATH)/etc/profile.d/conda.sh && conda
......@@ -49,12 +50,12 @@ install: # install conda environment
'
upload: # upload project files from client to server
$(RSYNC) $(LOCAL_PROJECT_DIR)/ $(LOGIN):$(REMOTE_PROJECT_DIR)
$(RSYNC) $(foreach file,$(LOCAL_UPLOAD_FILES),$(LOCAL_PROJECT_DIR)/$(file)) $(LOGIN):$(REMOTE_PROJECT_DIR)
run: # execute main script on server
$(SSH) '\
cd $(REMOTE_PROJECT_DIR);\
$(CONDA) activate -p $(CONDA_ENV);\
$(CONDA) activate $(CONDA_ENV);\
python $(MAIN_SCRIPT);\
$(CONDA) deactivate;\
'
......@@ -70,7 +71,7 @@ log: # show the last lines of the log file
'
download: # download output files from server to client
$(RSYNC) $(LOGIN):$(REMOTE_PROJECT_DIR)/$(REMOTE_OUTPUT_FILES) $(LOCAL_PROJECT_DIR)/$(LOCAL_OUTPUT_DIR)
$(RSYNC) $(LOGIN):$(REMOTE_PROJECT_DIR)/$(REMOTE_DOWNLOAD_FILES) $(LOCAL_PROJECT_DIR)/$(LOCAL_DOWNLOAD_DIR)
delete: # remove/delete all project files on server
$(SSH) '\
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment