SD card burn fails

[robertmoni]

Running dts init_sd_card with options or without gives the same result.

OS: ubuntu 18
Packages: all at their latest version
Inserted sd card can be found at /dev/sda. If the card is not inserted that drive is not shown.

DEBUG:dts:encoding: stdout UTF-8 stderr UTF-8 locale UTF-8.

dts :  Problems with a command?
    :  
    :  Report here: https://github.com/duckietown/duckietown-shell-commands/issues
    :  
    :  Troubleshooting:
    :  
    :  - If some commands update fail, delete ~/.dt-shell/commands
    :  
    :  - To reset the shell to "factory settings", delete ~/.dt-shell
    :  
    :    (Note: you will have to re-configure.)
INFO:dts:duckietown-shell-commands 4.0.18
INFO:duckietown-challenges:duckietown-challenges 4.0.30


## Tips and tricks

### Multiple networks

    dts init_sd_card --wifi  network1:password1,network2:password2 --country US



### Steps

Without arguments the script performs the steps:

    flash
    expand
    mount
    setup
    unmount

You can use --steps to run only some of those:

    dts init_sd_card --steps expand,mount



    
DEBUG:dts:Found 'wget' at /usr/bin/wget
DEBUG:dts:Found 'tar' at /bin/tar
DEBUG:dts:Found 'udisksctl' at /usr/bin/udisksctl
DEBUG:dts:Found 'docker' at /usr/bin/docker
DEBUG:dts:Found 'base64' at /usr/bin/base64
DEBUG:dts:Found 'gzip' at /bin/gzip
DEBUG:dts:Found 'udevadm' at /sbin/udevadm
DEBUG:dts:Found 'lsblk' at /bin/lsblk
INFO:dts:Please type the device with your SD card. Please be careful to pick the right device and to include '/dev/'. Here's a list of the devices on your system:
Running /bin/bash /home/dt/.dt-shell/commands/init_sd_card/list_disks.sh
NAME                              TYPE   SIZE VENDOR
/dev/sda                          disk  29.1G Generic
/dev/nvme0n1                      disk 238.5G
Done!
Type the name of your device (include the '/dev' part):   /dev/sda
Running /bin/bash /home/dt/.dt-shell/commands/init_sd_card/init_sd_card2.sh
64-bit OS detected...
Prior etcher-cli install detected at /tmp/duckietown/etcher-cli, skipping...
HypriotOS image was previously downloaded to /tmp/duckietown/hypriotos-rpi-v1.9.0.img.zip, skipping...
Flashing Hypriot image /tmp/duckietown/hypriotos-rpi-v1.9.0.img.zip to disk /dev/sda
? This will erase the selected drive. Are you sure? Yes
Bad archive

dts :  Traceback (most recent call last):
    :    File "/home/dt/.local/lib/python3.6/site-packages/dt_shell/__init__.py", line 48, in cli_main
    :      cli_main_()
    :    File "/home/dt/.local/lib/python3.6/site-packages/dt_shell/__init__.py", line 126, in cli_main_
    :      shell.onecmd(cmdline)
    :    File "/usr/lib/python3.6/cmd.py", line 217, in onecmd
    :      return func(arg)
    :    File "/home/dt/.local/lib/python3.6/site-packages/dt_shell/cli.py", line 410, in <lambda>
    :      do_command_lam = lambda s, w: do_command(klass, s, w)
    :    File "/home/dt/.local/lib/python3.6/site-packages/dt_shell/dt_command_abs.py", line 53, in do_command
    :      cls.command(shell, args)
    :    File "/home/dt/.dt-shell/commands/init_sd_card/command.py", line 204, in command
    :      step2function[step_name](shell, parsed)
    :    File "/home/dt/.dt-shell/commands/init_sd_card/command.py", line 273, in step_flash
    :      start_command_in_subprocess(script_cmd, env)
    :    File "/home/dt/.dt-shell/commands/utils/cli_utils.py", line 30, in start_command_in_subprocess
    :      raise Exception(msg)
    :  Exception: Error occurred while running /bin/bash /home/dt/.dt-shell/commands/init_sd_card/init_sd_card2.sh, please check and retry (1).

dts :  If you think this is a bug, please report that you are using:
    :  
    :  duckietown-shell-commands: 4.0.18
    :  duckietown-challenges: 4.0.30
    :  python: "3.6.7 (default, Oct 22 2018, 11:32:17) \n[GCC 8.2.0]"
    :  duckietown-shell: 4.0.22
    :  encodings:
    :    stdout: UTF-8
    :    stderr: UTF-8
    :    locale: UTF-8

[breandan]

What happens if you run rm /tmp/duckietown/hypriotos-rpi-v1.9.0.img.zip and retry?

[robertmoni]

Same exception error if I delete only the hypriotos zip file.
Deleting the whole duckietown folder from tmp resolves the issue ('rm -rf /tmp/duckietown').

Thanks for the hint @breandan.

[bemova]

I tried to burn the sd card. but, I received the "raise HTTPError(http_error_msg, response=self)
: requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.35/images/duckietown/rpi-simple-server:master18/json"

My docker is installed. and it works correctly.

Steps

Without arguments the script performs the steps:

flash
expand
mount
setup
unmount

You can use --steps to run only some of those:

dts init_sd_card --steps expand,mount

DEBUG:dts:Found 'wget' at /usr/bin/wget
DEBUG:dts:Found 'tar' at /bin/tar
DEBUG:dts:Found 'udisksctl' at /usr/bin/udisksctl
DEBUG:dts:Found 'docker' at /usr/bin/docker
DEBUG:dts:Found 'base64' at /usr/bin/base64
DEBUG:dts:Found 'gzip' at /bin/gzip
DEBUG:dts:Found 'udevadm' at /sbin/udevadm
DEBUG:dts:Found 'lsblk' at /bin/lsblk
INFO:dts:Please type the device with your SD card. Please be careful to pick the right device and to include '/dev/'. Here's a list of the devices on your system:
Running /bin/bash /home/arz/.dt-shell/commands/init_sd_card/list_disks.sh
NAME TYPE SIZE VENDOR
/dev/sda disk 29.7G ATA
/dev/sdb disk 150G VMware,
Done!
Type the name of your device (include the '/dev' part): /dev/sda
Running /bin/bash /home/arz/.dt-shell/commands/init_sd_card/init_sd_card2.sh
64-bit OS detected...
Prior etcher-cli install detected at /tmp/duckietown/etcher-cli, skipping...
HypriotOS image was previously downloaded to /tmp/duckietown/hypriotos-rpi-v1.9.0.img.zip, skipping...
Flashing Hypriot image /tmp/duckietown/hypriotos-rpi-v1.9.0.img.zip to disk /dev/sda
? This will erase the selected drive. Are you sure? Yes
Flashing: 13.83 MB/s [========================] 100% eta 0s
Validating: 0 B/s [========================] 100% eta 0s

Checksums:

• /dev/sda: 3ab3f1eb0b0b4e04f9630035ba95f26046b9e104a65dd78f8bb4fa9ed24c0dbebc647e8aa119c0b60bd8ba4d28e739709befbdfceac45ec2416b53c1b0f93423
  Flashing Hypriot image succeeded.
  Done!
  DEBUG:dts:Found 'parted' at /sbin/parted
  DEBUG:dts:Found 'resize2fs' at /sbin/resize2fs
  DEBUG:dts:Found 'df' at /bin/df
  DEBUG:dts:Found 'umount' at /bin/umount
  INFO:dts:Found device /dev/sda.
  INFO:dts:Current status:
  DEBUG:dts:$ ['sudo', 'lsblk', '/dev/sda']
  NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
  sda 8:0 0 29.7G 0 disk
  ├─sda1 8:1 0 64M 0 part
  └─sda2 8:2 0 935M 0 part
  DEBUG:dts:$ ['sudo', 'parted', '-s', '/dev/sda', 'resizepart', '2', '100%']
  DEBUG:dts:$ ['sudo', 'e2fsck', '-f', '/dev/sda2']
  e2fsck 1.44.1 (24-Mar-2018)
  Pass 1: Checking inodes, blocks, and sizes
  Pass 2: Checking directory structure
  Pass 3: Checking directory connectivity
  Pass 4: Checking reference counts
  Pass 5: Checking group summary information
  root: 30693/239360 files (0.2% non-contiguous), 217214/239359 blocks
  DEBUG:dts:$ ['sudo', 'resize2fs', '/dev/sda2']
  resize2fs 1.44.1 (24-Mar-2018)
  Resizing the filesystem on /dev/sda2 to 7775104 (4k) blocks.
  The filesystem on /dev/sda2 is now 7775104 (4k) blocks long.

INFO:dts:Updated status:
DEBUG:dts:$ ['sudo', 'lsblk', '/dev/sda']
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 29.7G 0 disk
├─sda1 8:1 0 64M 0 part
└─sda2 8:2 0 29.7G 0 part
DEBUG:dts:$ ['sudo', 'udevadm', 'trigger']
DEBUG:dts:$ ['udisksctl', 'mount', '-b', '/dev/disk/by-label/HypriotOS']
Mounted /dev/sda1 at /media/arz/HypriotOS.
DEBUG:dts:$ ['sudo', 'udevadm', 'trigger']
DEBUG:dts:$ ['udisksctl', 'mount', '-b', '/dev/disk/by-label/root']
Mounted /dev/sda2 at /media/arz/root.
/home/arz/.dt-shell/commands/init_sd_card/command.py:361: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
user_data = yaml.load(open(user_data_file).read())
INFO:dts:Adding file /home/arz/.ssh/authorized_keys
INFO:dts:Adding file /home/arz/.dt-shell/config
INFO:dts:Adding file /secrets/tokens/dt1
INFO:dts:Adding file /data/stats/init_sd_card/README.txt
INFO:dts:Adding file /data/stats/init_sd_card/CHANGELOG
INFO:dts:Adding file /data/stats/init_sd_card/version
INFO:dts:Adding file /data/stats/init_sd_card/flash_time
INFO:dts:Adding file /data/stats/init_sd_card/flash_user
INFO:dts:Adding file /data/stats/init_sd_card/flash_machine
INFO:dts:Adding file /data/stats/init_sd_card/parameters/hostname
INFO:dts:Adding file /data/stats/init_sd_card/parameters/linux_username
INFO:dts:Adding file /data/stats/init_sd_card/parameters/stacks_to_run
INFO:dts:Adding file /data/stats/init_sd_card/parameters/stacks_to_load
INFO:dts:Adding file /data/stats/init_sd_card/parameters/compress
INFO:dts:Adding file /data/stats/init_sd_card/parameters/device
INFO:dts:Adding file /data/stats/init_sd_card/parameters/country
INFO:dts:Adding file /data/stats/init_sd_card/parameters/wifi
INFO:dts:Adding file /data/stats/init_sd_card/parameters/ethz_username
INFO:dts:Adding file /data/stats/MAC/README.txt
INFO:dts:Configuration already found in ~/.ssh/config
INFO:dts:Adding file /var/local/wificfg.json
INFO:dts:Adding file /etc/wpa_supplicant/wpa_supplicant.conf
/home/arz/.dt-shell/commands/init_sd_card/command.py:744: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
add_file(path='/data/config/calibrations/kinematics/default.yaml', content=yaml.dump(yaml.load(kin_calib_file), default_flow_style=False))
INFO:dts:Adding file /data/config/calibrations/kinematics/default.yaml
/home/arz/.dt-shell/commands/init_sd_card/command.py:745: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
add_file(path='/data/config/calibrations/camera_extrinsic/default.yaml',content=yaml.dump(yaml.load(ext_cam_calib_file), default_flow_style=False))
INFO:dts:Adding file /data/config/calibrations/camera_extrinsic/default.yaml
/home/arz/.dt-shell/commands/init_sd_card/command.py:746: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
add_file(path='/data/config/calibrations/camera_intrinsic/default.yaml',content=yaml.dump(yaml.load(int_cam_calib_file), default_flow_style=False))
INFO:dts:Adding file /data/config/calibrations/camera_intrinsic/default.yaml
INFO:dts:Stacks to load: ['DT18_00_basic', 'DT18_01_health_stats', 'DT18_02_others', 'DT18_03_roscore', 'DT18_05_duckiebot_base', 'DT18_06_dashboard']
INFO:dts:Stacks to run: ['DT18_00_basic', 'DT18_01_health_stats', 'DT18_03_roscore', 'DT18_06_dashboard']
ERROR:dts:Could not find docker-compose. Cannot validate file.
INFO:dts:Adding file /var/local/DT18_00_basic.yaml
ERROR:dts:Could not find docker-compose. Cannot validate file.
INFO:dts:Adding file /var/local/DT18_01_health_stats.yaml
ERROR:dts:Could not find docker-compose. Cannot validate file.
INFO:dts:Adding file /var/local/DT18_02_others.yaml
ERROR:dts:Could not find docker-compose. Cannot validate file.
INFO:dts:Adding file /var/local/DT18_03_roscore.yaml
ERROR:dts:Could not find docker-compose. Cannot validate file.
INFO:dts:Adding file /var/local/DT18_05_duckiebot_base.yaml
ERROR:dts:Could not find docker-compose. Cannot validate file.
INFO:dts:Adding file /var/local/DT18_06_dashboard.yaml
INFO:dts:The stacks that are available are: DT18_03_roscore, DT18_02_others, DT18_05_duckiebot_base, DT18_04_camera, DT18_01_health_stats, DT18_99_swarm, DT18_06_dashboard, DT18_00_basic
INFO:dts:You asked to use ['DT18_00_basic', 'DT18_01_health_stats', 'DT18_02_others', 'DT18_03_roscore', 'DT18_05_duckiebot_base', 'DT18_06_dashboard']
/home/arz/.dt-shell/commands/init_sd_card/command.py:954: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
stacks2yaml[sn] = yaml.load(open(lpath).read())
INFO:dts:Pulling portainer/portainer:linux-arm
DEBUG:dts:$ ['docker', 'pull', 'portainer/portainer:linux-arm']
linux-arm: Pulling from portainer/portainer
Digest: sha256:ab4c6cabe6d2c2fbff48b9a00a99bc2b7a2a1fcb1b1c2e2e5cbc9372586c3f63
Status: Image is up to date for portainer/portainer:linux-arm
INFO:dts:Pulling v2tec/watchtower:armhf-latest
DEBUG:dts:$ ['docker', 'pull', 'v2tec/watchtower:armhf-latest']
armhf-latest: Pulling from v2tec/watchtower
Digest: sha256:63abf3fc71a46b8fc46674342766d976e6a97d6c70d2560a0986ec63c1784812
Status: Image is up to date for v2tec/watchtower:armhf-latest
INFO:dts:Using cached file /tmp/duckietown/docker_images/DT18_00_basic-d10d8dc0.tar
INFO:dts:Pulling duckietown/rpi-simple-server:master18
DEBUG:dts:$ ['docker', 'pull', 'duckietown/rpi-simple-server:master18']
master18: Pulling from duckietown/rpi-simple-server
Digest: sha256:6b6787319cd87b413d5b0a8737f6127c719293c12936c94b5ea3a2d7fd62f6b5
Status: Image is up to date for duckietown/rpi-simple-server:master18

dts : Traceback (most recent call last):
: File "/home/arz/.local/lib/python3.6/site-packages/docker/api/client.py", line 261, in raise_for_status
: response.raise_for_status()
: File "/home/arz/.local/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
: raise HTTPError(http_error_msg, response=self)
: requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.35/images/duckietown/rpi-simple-server:master18/json
:
: During handling of the above exception, another exception occurred:
:
: Traceback (most recent call last):
: File "/home/arz/.local/lib/python3.6/site-packages/dt_shell/init.py", line 48, in cli_main
: cli_main()
: File "/home/arz/.local/lib/python3.6/site-packages/dt_shell/init.py", line 126, in cli_main_
: shell.onecmd(cmdline)
: File "/usr/lib/python3.6/cmd.py", line 217, in onecmd
: return func(arg)
: File "/home/arz/.local/lib/python3.6/site-packages/dt_shell/cli.py", line 410, in
: do_command_lam = lambda s, w: do_command(klass, s, w)
: File "/home/arz/.local/lib/python3.6/site-packages/dt_shell/dt_command_abs.py", line 47, in do_command
: cls.command(shell, args)
: File "/home/arz/.dt-shell/commands/init_sd_card/command.py", line 204, in command
: step2function[step_name](shell, parsed)
: File "/home/arz/.dt-shell/commands/init_sd_card/command.py", line 463, in step_setup
: configure_images(parsed, user_data, add_file_local, add_file)
: File "/home/arz/.dt-shell/commands/init_sd_card/command.py", line 535, in configure_images
: stack2info = save_images(stack2yaml, compress=parsed.compress)
: File "/home/arz/.dt-shell/commands/init_sd_card/command.py", line 881, in save_images
: image = client.images.get(image_name)
: File "/home/arz/.local/lib/python3.6/site-packages/docker/models/images.py", line 316, in get
: return self.prepare_model(self.client.api.inspect_image(name))
: File "/home/arz/.local/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
: return f(self, resource_id, *args, **kwargs)
: File "/home/arz/.local/lib/python3.6/site-packages/docker/api/image.py", line 245, in inspect_image
: self._get(self._url("/images/{0}/json", image)), True
: File "/home/arz/.local/lib/python3.6/site-packages/docker/api/client.py", line 267, in _result
: self._raise_for_status(response)
: File "/home/arz/.local/lib/python3.6/site-packages/docker/api/client.py", line 263, in _raise_for_status
: raise create_api_error_from_http_exception(e)
: File "/home/arz/.local/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
: raise cls(e, response=response, explanation=explanation)
: docker.errors.APIError: 500 Server Error: Internal Server Error ("readlink /var/lib/docker/overlay2: invalid argument")

dts : If you think this is a bug, please report that you are using:
:
: duckietown-shell-commands: 4.0.21
: duckietown-challenges: 4.0.30
: python: "3.6.7 (default, Oct 22 2018, 11:32:17) \n[GCC 8.2.0]"
: duckietown-shell: 4.0.22
: encodings:
: stdout: UTF-8
: stderr: UTF-8
: locale: UTF-8

[bemova]

My OS is : Ubuntu 18.04
I am running this os as a vmware virtual machine.
The host os is Windows 10.

[breandan]

Never tried flashing an SD card from a VM before, but I suspect that is your problem. Seems like your Docker client cannot access the resources it needs, maybe the VM is not configured properly to access the SD or HD? cf. aws/aws-sam-cli#959

…

On Fri, Jun 7, 2019 at 11:12 AM mj ***@***.***> wrote: My OS is : Ubuntu 18.04 I am running this os as a vmware virtual machine. The host os is Windows 10. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#48?email_source=notifications&email_token=AABK4ZHZAMXIFXN5OGL3NPLPZJ3G3A5CNFSM4HSDJQLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXGDXUY#issuecomment-499923923>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABK4ZE4E56A24EGF57GQHTPZJ3G3ANCNFSM4HSDJQLA> .

[mantlekey]

Hello guys, I'm having a similar problem burning the SD card. I'm using a Raspberry PI 3 and after I type the command the terminal gets stuck.

Any suggestions?