[BUG] Random Disconnects from Broadcastify / Other Icecast Outputs

[zigsphere]

Using a Raspberry PI 3 and using the armv7-generic platform.
2 RTLSDR devices connected to a USB hub (powered)

I'm using the latest version 4.2.0 of rtl_airband.
Raspbian GNU/Linux 12 (bookworm)

What happened?

I am getting random disconnects / reconnects to broadcastify and other icecast endpoints I am using. I am currently running 2 rtlsdr dongles and recently discoverd the multithreading - that is enabled. I am trying to understand why the disconnects happen. My internet has no issues - constantly pinging google as an example - no issues. Perhaps there is something I can adjust in my airband configuration? The connection drops 15 - 30 minutes after running. Load, CPU, memory are also fine on the PI.

What you expected to happen?
Run rtl_airband and have no disconnects to broadcastify / other platforms for outputs.

Steps to Reproduce
Start rtl_airband and just wait.

Additional context

     Loaded: loaded (/etc/systemd/system/rtl_airband.service; enabled;
 preset: enabled)
     Active: active (running) since Fri 2024-01-12 03:26:05 UTC; 57min
 ago
       Docs: https://github.com/charlie-foxtrot/RTLSDR-Airband/wiki
   Main PID: 1056 (rtl_airband)
      Tasks: 7 (limit: 1571)
        CPU: 40min 54.865s
     CGroup: /system.slice/rtl_airband.service
             └─1056 /usr/local/bin/rtl_airband -Fe

Jan 12 03:57:13 pi3 rtl_airband[1056]: Connecting to audio9.broadcasti
fy.com:80/XXXX...
Jan 12 03:57:14 pi3 rtl_airband[1056]: Connected to audio9.broadcastif
y.com:80/XXXX
Jan 12 04:07:10 pi3 rtl_airband[1056]: Lost connection to audio9.broad
castify.com:80/XXXX
Jan 12 04:07:14 pi3 rtl_airband[1056]: Trying to reconnect to audio9.b
roadcastify.com:80/XXXX...
Jan 12 04:07:14 pi3 rtl_airband[1056]: Connecting to audio9.broadcasti
fy.com:80/XXXX...
Jan 12 04:07:15 pi3 rtl_airband[1056]: Connected to audio9.broadcastif
y.com:80/XXXX
Jan 12 04:13:12 pi3 rtl_airband[1056]: Lost connection to audio1.broad
castify.com:80/XXXX
Jan 12 04:13:15 pi3 rtl_airband[1056]: Trying to reconnect to audio1.b
roadcastify.com:80/XXXX...
Jan 12 04:13:15 pi3 rtl_airband[1056]: Connecting to audio1.broadcasti
fy.com:80/XXXX...
Jan 12 04:13:16 pi3 rtl_airband[1056]: Connected to audio1.broadcastif
y.com:80/XXXX

Your rtl_airband.conf file

multiple_output_threads = true;
devices:
({
    type = "rtlsdr";
    index = 0;
    gain = 35;
    centerfreq = 121.520;
    correction = 0;
    sample_rate = 2.40;
    channels:
    (
      {
        freq = 122.800;
        modulation = "am";
        lowpass = -1;
        highpass = 300;
        label = "Roseburg Regional Airport (122.800)";
        bandwidth = 5000;
        ampfactor = 2.00;
        outputs: (
          {
            type = "icecast";
            server = "audio9.broadcastify.com";
            port = 80;
            mountpoint = "XXXX";
            name = "Roseburg Regional Airport Unicom/CTAF";
            genre = "Aviation";
            username = "source";
            password = "XXXX";
            },
          {
            type = "file";
            directory = "/tmp";
            filename_template = "airport";
          }
        );
      }
    );
  },
  {
    type = "rtlsdr";
    index = 1;
    gain = 49;
    centerfreq = 155.242;
    correction = 0;
    sample_rate = 2.40;
    channels:
    (
      {
        freq = 155.700;
        modulation = "nfm";
        #lowpass = 5000;
        highpass = 300;
        lowpass = -1;
        label = "Douglas County Sheriff Law 1 (155.700)";
        bandwidth = 5000;
        ctcss = 151.4;
        outputs: (
          {
            type = "file";
            directory = "/tmp";
            split_on_transmission = True;
            silence_release = 2.0;
            include_freq = true;
            filename_template = "law1";
          },
          {
            type = "icecast";
            server = "audio9.broadcastify.com";
            port = 80;
            mountpoint = "XXXX";
            name = "Douglas County Sheriff - Law 1";
            genre = "Public Safety";
            username = "source";
            password = "XXXX";
          }
        );
      },
      {
        freq = 155.550;
        modulation = "nfm";
        lowpass = -1;
        highpass = 300;
        label = "Douglas County Sheriff - Law 2";
        bandwidth = 5000;
        #ampfactor = 2.00;
        ctcss = 151.4;
        outputs: (
          {
            type = "file";
            directory = "/tmp";
            filename_template = "law2";
            split_on_transmission = True;
            silence_release = 2.0;
            include_freq = true;
          },
          {
            type = "icecast";
            server = "audio9.broadcastify.com";
            port = 80;
            mountpoint = "XXXX";
            name = "Douglas County Sheriff - Law 2";
            genre = "Public Safety";
            username = "source";
            password = "XXXX";
          }
        );
      },
      {
        freq = 155.775;
        modulation = "nfm";
        lowpass = -1;
        highpass = 300;
        label = "Oregon Department of Transportation District 7 (155.775)";
        bandwidth = 5000;
        ctcss = 100.0;
        outputs: (
          {
            type = "file";
            directory = "/tmp";
            filename_template = "odot";
            split_on_transmission = True;
            silence_release = 2.0;
            include_freq = true;
          },
          {
            type = "icecast";
            server = "audio9.broadcastify.com";
            port = 80;
            mountpoint = "XXXX";
            name = "Oregon Department of Transportation - District 7";
            genre = "Public Safety";
            username = "source";
            password = "XXXX";
          }
        );
      },
      {
        freq = 154.815;
        modulation = "nfm";
        lowpass = -1;
        highpass = 300;
        label = "Oregon State Police - Roseburg Station";
        bandwidth = 5000;
        ctcss = 131.8;
        outputs: (
          {
            type = "file";
            directory = "/tmp";
            filename_template = "osp";
            split_on_transmission = True;
            silence_release = 2.0;
            include_freq = true;
          },
          {
            type = "icecast";
            server = "audio9.broadcastify.com";
            port = 80;
            mountpoint = "XXXX";
            name = "Oregon State Police - Roseburg Station";
            genre = "Public Safety";
            username = "source";
            password = "XXXX";
          }
        );
      },
      {
        freq = 154.370;
        modulation = "nfm";
        lowpass = -1;
        highpass = 300;
        label = "Douglas County Fire Dispatch";
        bandwidth = 5000;
        ctcss = 151.4;
        outputs: (
          {
            type = "file";
            directory = "/tmp";
            filename_template = "fire";
            split_on_transmission = True;
            silence_release = 2.0;
            include_freq = true;
          },
          {
            type = "icecast";
            server = "audio1.broadcastify.com";
            port = 80;
            mountpoint = "XXXX";
            name = "Douglas County Fire Dispatch";
            genre = "Public Safety";
            username = "source";
            password = "XXXX";
          },
          {
            type = "icecast";
            server = "listen.theroseburgreceiver.com";
            port = 8015;
            mountpoint = "";
            name = "Douglas County Fire Dispatch - Direct";
            genre = "Public Safety";
            username = "source";
            password = "XXXX";
          }
        );
      }
    );
  }
 );

[charlie-foxtrot]

I have multiple units running both local icecast and broadcastify feeds and don't see this on a few that I checked. All of mine are running on pi 4s however, so I wonder if that is a factor . . .

In addition to multiple_output_threads = true you can also use multiple_demod_threads = true, this will run a different thread for each dongle and will help keep up with the raw data rate. I think on the pi 3 both ethernet and USB share the same bus, so perhaps a backlog there could cause issues?

If you recompile with native instead of armv7-generic that will tune to the hardware you're building on (presumably the pi itself), not sure if that will impact anything or not.

There are a bunch of stats captured and can be written to a file using stats_filepath = "/tmp/radio.prom", enabling that and watching for changes when the disconnects happen may be interesting.

[zigsphere]

Sounds good. Thank you. I also just received a new PI5. With that being the case, what would be the best platform to use when compiling? Ill give your suggestions a shot.

[charlie-foxtrot]

If you are compiling on the machine you will be running on, use native as the platform.

[zigsphere]

What I have noticed is a buffer overrun count:

# HELP channel_activity_counter Loops of output_thread with frequency active.
# TYPE channel_activity_counter counter
channel_activity_counter{freq="122.800",label="Roseburg Regional Airport (122.800)"}	0
channel_activity_counter{freq="155.700",label="Douglas County Sheriff Law 1 (155.700)"}	44
channel_activity_counter{freq="155.550",label="Douglas County Sheriff - Law 2"}	0
channel_activity_counter{freq="155.775",label="Oregon Department of Transportation District 7 (155.775)"}	0
channel_activity_counter{freq="154.815",label="Oregon State Police - Roseburg Station"}	0
channel_activity_counter{freq="154.370",label="Douglas County Fire Dispatch"}	0

# HELP channel_noise_level Raw squelch noise_level.
# TYPE channel_noise_level gauge
channel_noise_level{freq="122.800",label="Roseburg Regional Airport (122.800)"}	0.091
channel_noise_level{freq="155.700",label="Douglas County Sheriff Law 1 (155.700)"}	0.286
channel_noise_level{freq="155.550",label="Douglas County Sheriff - Law 2"}	0.286
channel_noise_level{freq="155.775",label="Oregon Department of Transportation District 7 (155.775)"}	0.285
channel_noise_level{freq="154.815",label="Oregon State Police - Roseburg Station"}	0.283
channel_noise_level{freq="154.370",label="Douglas County Fire Dispatch"}	0.250

# HELP channel_dbfs_noise_level Squelch noise_level as dBFS.
# TYPE channel_dbfs_noise_level gauge
channel_dbfs_noise_level{freq="122.800",label="Roseburg Regional Airport (122.800)"}	-48.739
channel_dbfs_noise_level{freq="155.700",label="Douglas County Sheriff Law 1 (155.700)"}	-38.829
channel_dbfs_noise_level{freq="155.550",label="Douglas County Sheriff - Law 2"}	-38.826
channel_dbfs_noise_level{freq="155.775",label="Oregon Department of Transportation District 7 (155.775)"}	-38.870
channel_dbfs_noise_level{freq="154.815",label="Oregon State Police - Roseburg Station"}	-38.915
channel_dbfs_noise_level{freq="154.370",label="Douglas County Fire Dispatch"}	-39.979

# HELP channel_signal_level Raw squelch signal_level.
# TYPE channel_signal_level gauge
channel_signal_level{freq="122.800",label="Roseburg Regional Airport (122.800)"}	0.109
channel_signal_level{freq="155.700",label="Douglas County Sheriff Law 1 (155.700)"}	0.300
channel_signal_level{freq="155.550",label="Douglas County Sheriff - Law 2"}	0.317
channel_signal_level{freq="155.775",label="Oregon Department of Transportation District 7 (155.775)"}	0.309
channel_signal_level{freq="154.815",label="Oregon State Police - Roseburg Station"}	0.320
channel_signal_level{freq="154.370",label="Douglas County Fire Dispatch"}	0.300

# HELP channel_dbfs_signal_level Squelch signal_level as dBFS.
# TYPE channel_dbfs_signal_level gauge
channel_dbfs_signal_level{freq="122.800",label="Roseburg Regional Airport (122.800)"}	-47.194
channel_dbfs_signal_level{freq="155.700",label="Douglas County Sheriff Law 1 (155.700)"}	-38.404
channel_dbfs_signal_level{freq="155.550",label="Douglas County Sheriff - Law 2"}	-37.945
channel_dbfs_signal_level{freq="155.775",label="Oregon Department of Transportation District 7 (155.775)"}	-38.159
channel_dbfs_signal_level{freq="154.815",label="Oregon State Police - Roseburg Station"}	-37.853
channel_dbfs_signal_level{freq="154.370",label="Douglas County Fire Dispatch"}	-38.412

# HELP channel_squelch_counter Squelch open_count.
# TYPE channel_squelch_counter counter
channel_squelch_counter{freq="122.800",label="Roseburg Regional Airport (122.800)"}	0
channel_squelch_counter{freq="155.700",label="Douglas County Sheriff Law 1 (155.700)"}	1
channel_squelch_counter{freq="155.550",label="Douglas County Sheriff - Law 2"}	0
channel_squelch_counter{freq="155.775",label="Oregon Department of Transportation District 7 (155.775)"}	0
channel_squelch_counter{freq="154.815",label="Oregon State Police - Roseburg Station"}	0
channel_squelch_counter{freq="154.370",label="Douglas County Fire Dispatch"}	0

# HELP channel_squelch_level Squelch squelch_level.
# TYPE channel_squelch_level gauge
channel_squelch_level{freq="122.800",label="Roseburg Regional Airport (122.800)"}	0.274
channel_squelch_level{freq="155.700",label="Douglas County Sheriff Law 1 (155.700)"}	0.904
channel_squelch_level{freq="155.550",label="Douglas County Sheriff - Law 2"}	0.904
channel_squelch_level{freq="155.775",label="Oregon Department of Transportation District 7 (155.775)"}	0.900
channel_squelch_level{freq="154.815",label="Oregon State Police - Roseburg Station"}	0.895
channel_squelch_level{freq="154.370",label="Douglas County Fire Dispatch"}	0.792

# HELP channel_flappy_counter Squelch flappy_count.
# TYPE channel_flappy_counter counter
channel_flappy_counter{freq="122.800",label="Roseburg Regional Airport (122.800)"}	0
channel_flappy_counter{freq="155.700",label="Douglas County Sheriff Law 1 (155.700)"}	0
channel_flappy_counter{freq="155.550",label="Douglas County Sheriff - Law 2"}	0
channel_flappy_counter{freq="155.775",label="Oregon Department of Transportation District 7 (155.775)"}	0
channel_flappy_counter{freq="154.815",label="Oregon State Police - Roseburg Station"}	0
channel_flappy_counter{freq="154.370",label="Douglas County Fire Dispatch"}	0

# HELP channel_ctcss_counter count of windows with CTCSS detected.
# TYPE channel_ctcss_counter counter
channel_ctcss_counter{freq="122.800",label="Roseburg Regional Airport (122.800)"}	0
channel_ctcss_counter{freq="155.700",label="Douglas County Sheriff Law 1 (155.700)"}	13
channel_ctcss_counter{freq="155.550",label="Douglas County Sheriff - Law 2"}	0
channel_ctcss_counter{freq="155.775",label="Oregon Department of Transportation District 7 (155.775)"}	0
channel_ctcss_counter{freq="154.815",label="Oregon State Police - Roseburg Station"}	0
channel_ctcss_counter{freq="154.370",label="Douglas County Fire Dispatch"}	0

# HELP channel_no_ctcss_counter count of windows without CTCSS detected.
# TYPE channel_no_ctcss_counter counter
channel_no_ctcss_counter{freq="122.800",label="Roseburg Regional Airport (122.800)"}	0
channel_no_ctcss_counter{freq="155.700",label="Douglas County Sheriff Law 1 (155.700)"}	0
channel_no_ctcss_counter{freq="155.550",label="Douglas County Sheriff - Law 2"}	0
channel_no_ctcss_counter{freq="155.775",label="Oregon Department of Transportation District 7 (155.775)"}	0
channel_no_ctcss_counter{freq="154.815",label="Oregon State Police - Roseburg Station"}	0
channel_no_ctcss_counter{freq="154.370",label="Douglas County Fire Dispatch"}	0

# HELP buffer_overflow_count Number of times a device's buffer has overflowed.
# TYPE buffer_overflow_count counter
buffer_overflow_count{device="0"}	0
buffer_overflow_count{device="1"}	0

# HELP output_overrun_count Number of times a device or mixer output has overrun.
# TYPE output_overrun_count counter
output_overrun_count{device="0"}	0
output_overrun_count{device="1"}	9

The bandwidth for my feeds are set to 5000. Is that too big? There are only two dongles currently, but I plan to add a third.

Also, just as an update, this is now running on the PI5. Values multiple_output_threads = true and multiple_demod_threads = true have been set. Would the overrun count cause this?

When the drop occurred on icecast, I saw:

# HELP output_overrun_count Number of times a device or mixer output has overrun.
# TYPE output_overrun_count counter
output_overrun_count{device="0"}	0
output_overrun_count{device="1"}	48

Perhaps this is the culprit, but Im trying to understand why.

Thanks