Issue with users deleted and then recreated

[pierrepo]

Every academic year, we delete old student accounts and create new ones. Logins are the same and are recycled from one session to the next (stu-megm1-01, stu-megm1-02...)

We noticed an issue when account are re-created. For instance, with the account stu-megm1-50:

$ ls -al /srv/home/stu-megm1-50
total 32
drwxr-xr-x  5 stu-megm1-50 stu-megm1-50 4096 sept.  5 20:39 .
drwxr-xr-x 82 root         root         4096 sept.  3 16:48 ..
-rw-r--r--  1 stu-megm1-50 stu-megm1-50  220 févr. 25  2020 .bash_logout
-rw-r--r--  1 stu-megm1-50 stu-megm1-50 3771 févr. 25  2020 .bashrc
drwxrwxr-x  3 stu-megm1-50 stu-megm1-50 4096 sept.  5 20:39 .cache
drwxrwxr-x 12 stu-megm1-50 stu-megm1-50 4096 sept.  5 20:39 introduction_plasma
drwxrwxr-x  3 stu-megm1-50 stu-megm1-50 4096 sept.  5 20:39 .local
-rw-r--r--  1 stu-megm1-50 stu-megm1-50  807 févr. 25  2020 .profile

The user stu-megm1-50 opened the environment introduction_plasma and the corresponding directory has been created in his home directory with proper rights and owner.

After this account has been deleted and recreated, the user stu-megm1-50 is not able to load the same environment:

The error message is

Error: HTTP 500: Internal Server Error (Spawner failed to start [status=ExitCode=1, Error='', FinishedAt=2021-09-05T20:58:56.666401661Z]. The logs for stu-megm1-50 may contain details.)

If we look at his home directory:

$ ls -al /srv/home/stu-megm1-50
total 24
drwxr-xr-x  3 stu-megm1-50 stu-megm1-50 4096 sept.  5 20:58 .
drwxr-xr-x 82 root         root         4096 sept.  5 20:55 ..
-rw-r--r--  1 stu-megm1-50 stu-megm1-50  220 févr. 25  2020 .bash_logout
-rw-r--r--  1 stu-megm1-50 stu-megm1-50 3771 févr. 25  2020 .bashrc
drwxrwxr-x 12 stu-megm1-49 stu-megm1-49 4096 sept.  5 20:58 introduction_plasma
-rw-r--r--  1 stu-megm1-50 stu-megm1-50  807 févr. 25  2020 .profile

The directory introduction_plasma corresponding to the selected environment has been created, but with the wrong owner (stu-megm1-49 instead of stu-megm1-50), thus preventing Jupyter Lab to be correctly loaded with this environment.

It looks like a memory of old UIG/GID is kept and improperly used with recycled logins.

[jtpio]

@pierrepo are you able to reproduce this with test users?

Also were the users created with the procedure in https://docs.plasmabio.org/en/latest/install/users.html?
Or from a CSV file? https://docs.plasmabio.org/en/latest/configuration/batch-users.html#batch-users

Normally the permissions of the home subdirectory are set on startup here:

plasma/tljh-plasma/tljh_plasma/entrypoint/entrypoint.sh

Lines 34 to 35 in a74d7a0

	# set the correct permissions for the user home subdirectory
	chown -R ${NB_USER}:${NB_USER} ${IMAGE_DIR}

[pierrepo]

As far as I remember, I got this error only when creating users by batch (from a CSV file).

[pierrepo]

I've created/deleted/recreate users on a fresh new VM.

Step 1. Create 5 new users in batch: student1 to student5

All users and home folders are properly created:

$ ll /srv/home/
total 36
drwxr-xr-x 9 root     root     4096 Apr  7 20:22 ./
drwxr-xr-x 4 root     root     4096 Apr  7 08:19 ../
drwxr-xr-x 2 bar      bar      4096 Apr  7 07:25 bar/
drwxr-xr-x 7 foo      foo      4096 Apr  7 20:00 foo/
drwxr-xr-x 7 student1 student1 4096 Apr  7 20:23 student1/
drwxr-xr-x 7 student2 student2 4096 Apr  7 20:24 student2/
drwxr-xr-x 7 student3 student3 4096 Apr  7 20:25 student3/
drwxr-xr-x 2 student4 student4 4096 Apr  7 20:22 student4/
drwxr-xr-x 2 student5 student5 4096 Apr  7 20:22 student5/

For the record, user ids are:

ubuntu@plasmabio-test:/srv/home/foo$ id student1
uid=1003(student1) gid=1009(student1) groups=1009(student1),1004(group_1)
ubuntu@plasmabio-test:/srv/home/foo$ id student2
uid=1004(student2) gid=1010(student2) groups=1010(student2),1004(group_1)
ubuntu@plasmabio-test:/srv/home/foo$ id student3
uid=1005(student3) gid=1011(student3) groups=1011(student3),1004(group_1)
ubuntu@plasmabio-test:/srv/home/foo$ id student4
uid=1006(student4) gid=1012(student4) groups=1012(student4),1004(group_1)
ubuntu@plasmabio-test:/srv/home/foo$ id student5
uid=1007(student5) gid=1013(student5) groups=1013(student5),1004(group_1)

Step 2. Remove users student1 to student5

Users and home folders are deleted, as expected:

$ ls -l /srv/home/
total 8
drwxr-xr-x 2 bar bar 4096 Apr  7 07:25 bar
drwxr-xr-x 7 foo foo 4096 Apr  7 20:00 foo

Step 3. Create 3 new users: student6 to student8

All users and home folders are properly created:

$ ls -l /srv/home/
total 20
drwxr-xr-x 2 bar      bar      4096 Apr  7 07:25 bar
drwxr-xr-x 7 foo      foo      4096 Apr  7 20:00 foo
drwxr-xr-x 2 student6 student6 4096 Apr  8 16:12 student6
drwxr-xr-x 2 student7 student7 4096 Apr  8 16:12 student7
drwxr-xr-x 2 student8 student8 4096 Apr  8 16:12 student8

For the record, user ids are:

ubuntu@plasmabio-test:~$ id student6
uid=1003(student6) gid=1009(student6) groups=1009(student6),1004(group_1)
ubuntu@plasmabio-test:~$ id student7
uid=1004(student7) gid=1010(student7) groups=1010(student7),1004(group_1)
ubuntu@plasmabio-test:~$ id student8
uid=1005(student8) gid=1011(student8) groups=1011(student8),1004(group_1)

User ids are recycled from previously deleted users. This looks fine.

Step 4. Recreate old users student1 to student5

All users and home folders are properly created:

$ ls -l /srv/home/
total 40
drwxr-xr-x 2 bar      bar      4096 Apr  7 07:25 bar
drwxr-xr-x 7 foo      foo      4096 Apr  7 20:00 foo
drwxr-xr-x 2 student1 student1 4096 Apr  8 16:12 student1
drwxr-xr-x 2 student2 student2 4096 Apr  8 16:12 student2
drwxr-xr-x 2 student3 student3 4096 Apr  8 16:12 student3
drwxr-xr-x 2 student4 student4 4096 Apr  8 16:12 student4
drwxr-xr-x 2 student5 student5 4096 Apr  8 16:12 student5
drwxr-xr-x 2 student6 student6 4096 Apr  8 16:12 student6
drwxr-xr-x 2 student7 student7 4096 Apr  8 16:12 student7
drwxr-xr-x 2 student8 student8 4096 Apr  8 16:12 student8

New user ids are assigned:

ubuntu@plasmabio-test:~$ id student1
uid=1006(student1) gid=1012(student1) groups=1012(student1),1004(group_1)
ubuntu@plasmabio-test:~$ id student2
uid=1007(student2) gid=1013(student2) groups=1013(student2),1004(group_1)
ubuntu@plasmabio-test:~$ id student3
uid=1008(student3) gid=1014(student3) groups=1014(student3),1004(group_1)
ubuntu@plasmabio-test:~$ id student4
uid=1009(student4) gid=1015(student4) groups=1015(student4),1004(group_1)
ubuntu@plasmabio-test:~$ id student5
uid=1010(student5) gid=1016(student5) groups=1016(student5),1004(group_1)

Step 5. Connect as student1 and run an environment

We connect with student1 account and run an environment. Connection is fine but we have an error:

Looking at student1's user folder, we have a folder that does not belong to him:

$ ls -l /srv/home/student1
total 4
drwxrwxr-x 6 student6 plasma-users 4096 Apr  8 16:13 plasmabio-template-python

student1 user id is now 1006. His former id (1003) has been assigned to student6. It looks like Jupyter (Hub/Server?) has in memory the old user id of student1. This is unexpected and prevent the environment to launch.

From the server, we can get this log:

$ sudo journalctl CONTAINER_NAME=jupyter-student1-
Apr 08 16:13:55 plasmabio-test c624049683df[13526]: Traceback (most recent call last):
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/traitlets/traitlets.py", line 537, in get
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     value = obj._trait_values[self.name]
Apr 08 16:13:55 plasmabio-test c624049683df[13526]: KeyError: 'runtime_dir'
Apr 08 16:13:55 plasmabio-test c624049683df[13526]: 
Apr 08 16:13:55 plasmabio-test c624049683df[13526]: During handling of the above exception, another exception occurred:
Apr 08 16:13:55 plasmabio-test c624049683df[13526]: 
Apr 08 16:13:55 plasmabio-test c624049683df[13526]: Traceback (most recent call last):
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/bin/jupyterhub-singleuser", line 10, in <module>
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     sys.exit(main())
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/jupyter_core/application.py", line 264, in launch_instance
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/traitlets/config/application.py", line 845, in launch_instance
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     app.initialize(argv)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/jupyterhub/singleuser/mixins.py", line 852, in initialize
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     result = super().initialize(*args, **kwargs)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/jupyterhub/singleuser/mixins.py", line 573, in initialize
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     return super().initialize(argv)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/traitlets/config/application.py", line 88, in inner
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     return method(app, *args, **kwargs)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/notebook/notebookapp.py", line 2127, in initialize
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     self.init_configurables()
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/notebook/notebookapp.py", line 1650, in init_configurables
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     connection_dir=self.runtime_dir,
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/traitlets/traitlets.py", line 577, in __get__
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     return self.get(obj, cls)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/traitlets/traitlets.py", line 540, in get
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     default = obj.trait_defaults(self.name)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/traitlets/traitlets.py", line 1580, in trait_defaults
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     return self._get_trait_default_generator(names[0])(self)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/jupyter_core/application.py", line 95, in _runtime_dir_default
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     ensure_dir_exists(rd, mode=0o700)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/site-packages/jupyter_core/utils/__init__.py", line 11, in ensure_dir_exists
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     os.makedirs(path, mode=mode)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/os.py", line 215, in makedirs
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     makedirs(head, exist_ok=exist_ok)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/os.py", line 215, in makedirs
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     makedirs(head, exist_ok=exist_ok)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/os.py", line 215, in makedirs
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     makedirs(head, exist_ok=exist_ok)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:   File "/srv/conda/envs/notebook/lib/python3.10/os.py", line 225, in makedirs
Apr 08 16:13:55 plasmabio-test c624049683df[13526]:     mkdir(name, mode)
Apr 08 16:13:55 plasmabio-test c624049683df[13526]: PermissionError: [Errno 13] Permission denied: '/srv/home/student1/.local'