Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue with users deleted and then recreated #189

Open
pierrepo opened this issue Sep 5, 2021 · 3 comments · May be fixed by #207
Open

Issue with users deleted and then recreated #189

pierrepo opened this issue Sep 5, 2021 · 3 comments · May be fixed by #207

Comments

@pierrepo
Copy link
Contributor

pierrepo commented Sep 5, 2021

Every academic year, we delete old student accounts and create new ones. Logins are the same and are recycled from one session to the next (stu-megm1-01, stu-megm1-02...)

We noticed an issue when account are re-created. For instance, with the account stu-megm1-50:

$ ls -al /srv/home/stu-megm1-50
total 32
drwxr-xr-x  5 stu-megm1-50 stu-megm1-50 4096 sept.  5 20:39 .
drwxr-xr-x 82 root         root         4096 sept.  3 16:48 ..
-rw-r--r--  1 stu-megm1-50 stu-megm1-50  220 févr. 25  2020 .bash_logout
-rw-r--r--  1 stu-megm1-50 stu-megm1-50 3771 févr. 25  2020 .bashrc
drwxrwxr-x  3 stu-megm1-50 stu-megm1-50 4096 sept.  5 20:39 .cache
drwxrwxr-x 12 stu-megm1-50 stu-megm1-50 4096 sept.  5 20:39 introduction_plasma
drwxrwxr-x  3 stu-megm1-50 stu-megm1-50 4096 sept.  5 20:39 .local
-rw-r--r--  1 stu-megm1-50 stu-megm1-50  807 févr. 25  2020 .profile

The user stu-megm1-50 opened the environment introduction_plasma and the corresponding directory has been created in his home directory with proper rights and owner.

After this account has been deleted and recreated, the user stu-megm1-50 is not able to load the same environment:

image

The error message is

Error: HTTP 500: Internal Server Error (Spawner failed to start [status=ExitCode=1, Error='', FinishedAt=2021-09-05T20:58:56.666401661Z]. The logs for stu-megm1-50 may contain details.)

If we look at his home directory:

$ ls -al /srv/home/stu-megm1-50
total 24
drwxr-xr-x  3 stu-megm1-50 stu-megm1-50 4096 sept.  5 20:58 .
drwxr-xr-x 82 root         root         4096 sept.  5 20:55 ..
-rw-r--r--  1 stu-megm1-50 stu-megm1-50  220 févr. 25  2020 .bash_logout
-rw-r--r--  1 stu-megm1-50 stu-megm1-50 3771 févr. 25  2020 .bashrc
drwxrwxr-x 12 stu-megm1-49 stu-megm1-49 4096 sept.  5 20:58 introduction_plasma
-rw-r--r--  1 stu-megm1-50 stu-megm1-50  807 févr. 25  2020 .profile

The directory introduction_plasma corresponding to the selected environment has been created, but with the wrong owner (stu-megm1-49 instead of stu-megm1-50), thus preventing Jupyter Lab to be correctly loaded with this environment.

It looks like a memory of old UIG/GID is kept and improperly used with recycled logins.
@jtpio
Copy link
Member

jtpio commented Mar 29, 2022

@pierrepo are you able to reproduce this with test users?

Also were the users created with the procedure in https://docs.plasmabio.org/en/latest/install/users.html?
Or from a CSV file? https://docs.plasmabio.org/en/latest/configuration/batch-users.html#batch-users

Normally the permissions of the home subdirectory are set on startup here:

plasma/tljh-plasma/tljh_plasma/entrypoint/entrypoint.sh

Lines 34 to 35 in a74d7a0

# set the correct permissions for the user home subdirectory
chown -R ${NB_USER}:${NB_USER} ${IMAGE_DIR}

@jtpio jtpio linked a pull request Mar 29, 2022 that will close this issue
Draft
1 task
@pierrepo
Copy link
Contributor Author

As far as I remember, I got this error only when creating users by batch (from a CSV file).

@pierrepo
Copy link
Contributor Author

I've created/deleted/recreate users on a fresh new VM.

Step 1. Create 5 new users in batch: student1 to student5

All users and home folders are properly created:

$ ll /srv/home/
total 36
drwxr-xr-x 9 root     root     4096 Apr  7 20:22 ./
drwxr-xr-x 4 root     root     4096 Apr  7 08:19 ../
drwxr-xr-x 2 bar      bar      4096 Apr  7 07:25 bar/
drwxr-xr-x 7 foo      foo      4096 Apr  7 20:00 foo/
drwxr-xr-x 7 student1 student1 4096 Apr  7 20:23 student1/
drwxr-xr-x 7 student2 student2 4096 Apr  7 20:24 student2/
drwxr-xr-x 7 student3 student3 4096 Apr  7 20:25 student3/
drwxr-xr-x 2 student4 student4 4096 Apr  7 20:22 student4/
drwxr-xr-x 2 student5 student5 4096 Apr  7 20:22 student5/

For the record, user ids are:

ubuntu@plasmabio-test:/srv/home/foo$ id student1
uid=1003(student1) gid=1009(student1) groups=1009(student1),1004(group_1)
ubuntu@plasmabio-test:/srv/home/foo$ id student2
uid=1004(student2) gid=1010(student2) groups=1010(student2),1004(group_1)
ubuntu@plasmabio-test:/srv/home/foo$ id student3
uid=1005(student3) gid=1011(student3) groups=1011(student3),1004(group_1)
ubuntu@plasmabio-test:/srv/home/foo$ id student4
uid=1006(student4) gid=1012(student4) groups=1012(student4),1004(group_1)
ubuntu@plasmabio-test:/srv/home/foo$ id student5
uid=1007(student5) gid=1013(student5) groups=1013(student5),1004(group_1)

Step 2. Remove users student1 to student5

Users and home folders are deleted, as expected:

$ ls -l /srv/home/
total 8
drwxr-xr-x 2 bar bar 4096 Apr  7 07:25 bar
drwxr-xr-x 7 foo foo 4096 Apr  7 20:00 foo

Step 3. Create 3 new users: student6 to student8

All users and home folders are properly created:

$ ls -l /srv/home/
total 20
drwxr-xr-x 2 bar      bar      4096 Apr  7 07:25 bar
drwxr-xr-x 7 foo      foo      4096 Apr  7 20:00 foo
drwxr-xr-x 2 student6 student6 4096 Apr  8 16:12 student6
drwxr-xr-x 2 student7 student7 4096 Apr  8 16:12 student7
drwxr-xr-x 2 student8 student8 4096 Apr  8 16:12 student8

For the record, user ids are:

ubuntu@plasmabio-test:~$ id student6
uid=1003(student6) gid=1009(student6) groups=1009(student6),1004(group_1)
ubuntu@plasmabio-test:~$ id student7
uid=1004(student7) gid=1010(student7) groups=1010(student7),1004(group_1)
ubuntu@plasmabio-test:~$ id student8
uid=1005(student8) gid=1011(student8) groups=1011(student8),1004(group_1)

User ids are recycled from previously deleted users. This looks fine.

Step 4. Recreate old users student1 to student5

All users and home folders are properly created:

$ ls -l /srv/home/
total 40
drwxr-xr-x 2 bar      bar      4096 Apr  7 07:25 bar
drwxr-xr-x 7 foo      foo      4096 Apr  7 20:00 foo
drwxr-xr-x 2 student1 student1 4096 Apr  8 16:12 student1
drwxr-xr-x 2 student2 student2 4096 Apr  8 16:12 student2
drwxr-xr-x 2 student3 student3 4096 Apr  8 16:12 student3
drwxr-xr-x 2 student4 student4 4096 Apr  8 16:12 student4
drwxr-xr-x 2 student5 student5 4096 Apr  8 16:12 student5
drwxr-xr-x 2 student6 student6 4096 Apr  8 16:12 student6
drwxr-xr-x 2 student7 student7 4096 Apr  8 16:12 student7
drwxr-xr-x 2 student8 student8 4096 Apr  8 16:12 student8

New user ids are assigned:

ubuntu@plasmabio-test:~$ id student1
uid=1006(student1) gid=1012(student1) groups=1012(student1),1004(group_1)
ubuntu@plasmabio-test:~$ id student2
uid=1007(student2) gid=1013(student2) groups=1013(student2),1004(group_1)
ubuntu@plasmabio-test:~$ id student3
uid=1008(student3) gid=1014(student3) groups=1014(student3),1004(group_1)
ubuntu@plasmabio-test:~$ id student4
uid=1009(student4) gid=1015(student4) groups=1015(student4),1004(group_1)
ubuntu@plasmabio-test:~$ id student5
uid=1010(student5) gid=1016(student5) groups=1016(student5),1004(group_1)

Step 5. Connect as student1 and run an environment

We connect with student1 account and run an environment. Connection is fine but we have an error:

image

Looking at student1's user folder, we have a folder that does not belong to him:

$ ls -l /srv/home/student1
total 4
drwxrwxr-x 6 student6 plasma-users 4096 Apr  8 16:13 plasmabio-template-python

student1 user id is now 1006. His former id (1003) has been assigned to student6. It looks like Jupyter (Hub/Server?) has in memory the old user id of student1. This is unexpected and prevent the environment to launch.

From the server, we can get this log:

$ sudo journalctl CONTAINER_NAME=jupyter-student1-
Apr 08 16:13:55 plasmabio-test c624049683df[13526]: Traceback (most recent call last):
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/traitlets/traitlets.py", line 537, in get
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     value = obj._trait_values[self.name]
Apr 08 16:13:55 plasmabio-test c624049683df[13526]: KeyError: 'runtime_dir'
Apr 08 16:13:55 plasmabio-test c624049683df[13526]: 
Apr 08 16:13:55 plasmabio-test c624049683df[13526]: During handling of the above exception, another exception occurred:
Apr 08 16:13:55 plasmabio-test c624049683df[13526]: 
Apr 08 16:13:55 plasmabio-test c624049683df[13526]: Traceback (most recent call last):
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/bin/jupyterhub-singleuser", line 10, in <module>
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     sys.exit(main())
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/jupyter_core/application.py", line 264, in launch_instance
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/traitlets/config/application.py", line 845, in launch_instance
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     app.initialize(argv)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/jupyterhub/singleuser/mixins.py", line 852, in initialize
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     result = super().initialize(*args, **kwargs)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/jupyterhub/singleuser/mixins.py", line 573, in initialize
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     return super().initialize(argv)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/traitlets/config/application.py", line 88, in inner
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     return method(app, *args, **kwargs)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/notebook/notebookapp.py", line 2127, in initialize
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     self.init_configurables()
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/notebook/notebookapp.py", line 1650, in init_configurables
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     connection_dir=self.runtime_dir,
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/traitlets/traitlets.py", line 577, in __get__
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     return self.get(obj, cls)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/traitlets/traitlets.py", line 540, in get
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     default = obj.trait_defaults(self.name)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/traitlets/traitlets.py", line 1580, in trait_defaults
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     return self._get_trait_default_generator(names[0])(self)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/jupyter_core/application.py", line 95, in _runtime_dir_default
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     ensure_dir_exists(rd, mode=0o700)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/jupyter_core/utils/__init__.py", line 11, in ensure_dir_exists
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     os.makedirs(path, mode=mode)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/os.py", line 215, in makedirs
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     makedirs(head, exist_ok=exist_ok)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/os.py", line 215, in makedirs
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     makedirs(head, exist_ok=exist_ok)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/os.py", line 215, in makedirs
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     makedirs(head, exist_ok=exist_ok)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/os.py", line 225, in makedirs
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     mkdir(name, mode)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]: PermissionError: [Errno 13] Permission denied: '/srv/home/student1/.local'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Ensure the user subdirectory has proper rights jtpio/plasma
2 participants
@jtpio @pierrepo