Lower case characters on mods downloaded from the Workhop

[CCVia]

I'm opening a couple of issues with two separate things I've found. Sorry about the spam!

After downloading a lot of mods from the Workwhop using the steam-workshop-mods-status branch some of them are not usable because they contain upper case characters. This is happening on Linux because of a known limitation in Linux servers (https://community.bistudio.com/wiki/Arma_3:_Dedicated_Server#Case_sensitivity_.26_Mods).

I don't know if it's possible to circumvent that issue with a solution like the one described at the link (using ciopfs). A lot of the mods I'm using contain upper case characters, and if I can't solve this issue I will need to revert to the main branch and keep updating everything manually. Maybe somebody else has found a way of using the steam-workshop-mods-status branch on Linux solving this on the server side. Any help would be really appreciated.

When using the main branch I just uploaded the mods and renamed everything with this script:

#!/bin/bash
# Rename all directories. This will need to be done first.
# Process each directory^rs contents before the directory itself
find * -depth -type d | while read x
do
        # Translate Caps to Small letters
        y=$(echo "$x" | tr '[A-Z ]' '[a-z_]');

        # create directory if it does not exit
        if [ ! -d "$y" ]; then
                mkdir -p "$y";
        fi

        # check if the source and destination is the same
        if [ "$x" != "$y" ]; then

                # move directory files before deleting
                ls -A "$x" | while read i
                do
                  mv "$x"/"$i" "$y";
                done
                rmdir "$x";

        fi

done

# Rename all files
find * -type f | while read x ;
do
        # Translate Caps to Small letters
        y=$(echo "$x" | tr '[A-Z ]' '[a-z_]');
        if [ "$x" != "$y" ]; then
                mv "$x" "$y";
        fi
done

exit 0

Thank you!

[Dahlgren]

You can use the rename tool (the perl version on debian/ubuntu) to do mass renaming with find $folder -depth -exec rename 's/(.*)\/([^\/]*)/$1\/\L$2/' {} \;

Ideally mod authors should package their mods with Linux support in mind as things such as native extensions needs to have matching call in sqf code for the .so file to work. Not all mods use lower case in native extension calling.

[CCVia]

Thank you @Dahlgren

Won't this break the updates?

I know, ideally people should have this restriction in mind, but I've found about 1/3 of the mods have upper case characters, so they probably never experienced the issue.

[Dahlgren]

Anything that makes changes to workshop files outside of steamcmd will cause issues on next invocation of steamcmd as the metadata and actual files will be out of sync.

For our (Anrop's) server infrastructure we have a mirror of Steam Workshop which is then mapped to proper lowercase folders and files via lndir. This avoids making changes to the steam files while keeping a mirror which does not eat any extra disk space for Arma. This mirror of Workshop files is then distributed out to the actual server hosts which use the main branch and not the steam workshop one.

[CCVia]

I don't know exactly how we do it in my team but we have a repository synced with ArmA3Sync that doesn't update automatically with Steam Workshop. Every mod that goes in there is checked and modified if needed so it works properly on the server, and we usually don't rush into updates until we're sure nothing will break. There we don't use your panel (or any panel at all), the missions and mods are uploaded via ssh and just 2 or 3 people have access. Old school...
The server I manage is an unofficial one we use for informal games, trying new mods, etc. Ideally it would be an easier to manage one, accessible to more people thanks to the web interface and less dependant on the admin. Hence the interest on the workshop-mods-status branch.

I don't know if it would be possible to implement a solution like yours but within the steam-workshop-mods-status branch. Maybe if the mods were loaded from within a mirror folder, not the original where they're downloaded, it might work. I've tried creating a mirror of the entire steam folder with ciopfs and then changing the "path-to-main-steam-folder" on config.js, but (of course) it didn't work. I guess might be possible to do something alike if I could choose from where are the mods loading into the panel.

[Dahlgren]

It would be possible with the changes made in the virtual server branch made for keys management. Thought it won't fix native extensions not being built with Linux in consideration. Although anyone publishing native extensions for Linux probably already considered that issue.

[CCVia]

I'm sorry but I thing you're like 1000 light-years ahead of me on this. I can manage a server but I lack a lot of knowledge necessary to solve this issue. Let me check if I'm following correctly; the changes made on the virtual-server branch, when/if merged into the workshop-mods-status branch, would allow to modify this branch in order to get the mods from a mirror directory, allowing to keep them synced via workshop API but avoiding the uppercase problem?

[Dahlgren]

Correct, it allows for each (running) server to make adjustments to things such as keys folder or to lower case addons folder and pbo files without changing the (source) mods

[CCVia]

Nice! But that version hasn't got the Workshop integration, right? Is there any chance the virtual-server gets merged with the workshop-mods-status branch?

[CCVia]

Just a quick update on how I've solved this issue (temporally). Maybe it can be useful to someone else.
I've switched back to the main branch and I've used a modified version of this script: https://gist.github.com/marceldev89/12da69b95d010c8a810fd384cca8d02a

#!/usr/bin/python3

# MIT License
#
# Copyright (c) 2017 Marcel de Vries
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

import os
import os.path
import re
import shutil
import time

from datetime import datetime
from urllib import request

#region Configuration
STEAM_CMD = "/home/arma3/steamcmd/steamcmd.sh"
STEAM_USER = "MY-USERNAME"
STEAM_PASS = "MY-PASSWORD"

A3_SERVER_ID = "233780"
A3_SERVER_DIR = "/home/arma3/arma3"
A3_WORKSHOP_ID = "107410"

A3_WORKSHOP_DIR = "/home/arma3/Steam/"
A3_MODS_DIR = "/home/arma3/arma3"

MODS = {

    "@3cb_baf_equipment":          	     "893328083",
    "@3cb_baf_units":            		 "893346105",
    "@3cb_baf_units_ace_compat":         "1135539579",
    "@3cb_baf_units_rhs_compat":         "1135541175",
    "@3cb_baf_vehicles":           		  "893349825",
    "@3cb_baf_weapons":           		  "893339590",
    "@3cb_baf_weapons_rhs_ammo_compat":                           "1515845502",
    "@3cb_baf_factions":        	     "1673456286",
    "@3den_enhanced":         		    "623475643",
    "@ace":            				   "463939057",
    "@ace_compat_rhs_afrf":             "773131200",
    "@ace_compat_rhs_usaf":             "773125288",
    "@ace_compat_rhs_gref":             "884966711",
    "@ace_compat_rhs_saf":             "2174495332",
    "@ace3_bwmod_compatibility":         "1200145989",
    "@acex":            			 "708250744",
    "@acre2":          			   "751965892",
    "@acre2_gm_compat":             "1740990569",
    "@aftermath":           	  "1285636607",
    "@aliabad_region":             "918639806",
    "@alive":            			 "620260972",
    "@archipelago":          	   "1726184748",
    "@arma_3_primal":        	     "1830220150",
    "@armstalker_mutants":             "1626948630",
    "@backpackonchest":             "820924072",
    "@battle_of_normandy_overlord_and_perch":             "2291305691",
    "@blastcore_edited_standalone_version":             "767380317",
    "@bozcaada":           	  "524622628",
    "@breaching_charge":             "1314910827",
    "@brush_clearing":             "1889104923",
    "@burn_em":           		  "1541221167",
    "@bwmod":           		  "1200127537",
    "@camofaces":           	  "346665985",
    "@cba_a3":           		  "450814997",
    "@cla_clafghan":              "761349672",
    "@cup_ace3_compat_terrains":             "1375890861",
    "@cup_ace3_compat_vehicles":             "621650475",
    "@cup_ace3_compat_weapons":             "549676314",
    "@cup_terrains_core": 				 "583496184",
    "@cup_terrains_maps": 				 "583544987",
    "@cup_terrains_maps20": 			 "1981964169",
    "@cup_units":         				 "497661914",
    "@cup_vehicles":     				  "541888371",
    "@cup_weapons":      				  "497660133",
    "@dhi_uniforms_and_equipment":          "1573550621",
    "@diwako_stalker_anomalies":             "1383903166",
    "@em_buildings":                          "671539540",
    "@enhanced_movement":          		   "333310405",
    "@ericj_taliban_units":             "668919514",
    "@expansion_mod_cbrn":         	    "1961695597",
    "@faces_of_war":            		 "891433622",
    "@faces_of_war_gold_beach":             "2043771912",
    "@fallujah_12":            			 "633789490",
    "@global_mobilization_extra_aaf":             "1816026173",
    "@global_mobilization_extra_takistan_af":             "1737435871",
    "@gm_enhancement_by_vpzbrig21":             "2283431514",
    "@hazar_kot_valley":           	  "766294759",
    "@head_range_plus_trackir":             "630737877",
    "@horror_mod":             "1556296528",
    "@ifa3_aio_lite":            		 "660460283",
    "@immersion_cigs":            		 "753946944",
    "@improved_melee_system":             "2291129343",
    "@isla_duala":         			    "714149065",
    "@jbad":          				   "520618345",
    "@jms_legion":                                       "2376768339",
    "@jsrs_soudmod":     		        "861133494",
    "@kholm":           			  "605959658",
    "@kujari":          			   "1726494027",
    "@kunduz_afghanistan_doors_multipl":             "1623903743",
    "@laghisola":         			    "2175069333",
    "@lambs_danger_fsm":             "1858075458",
    "@lambs_rpg":            			 "1858070328",
    "@lambs_supression":             "1808238502",
    "@lambs_turrets":        	     "1862208264",
    "@lythium":           				  "909547724",
    "@mbg_danish_m01_desert_camo_retex":             "2135236146",
    "@mrhmilsimtools":          	   "1196049538",
    "@nassau1715":          		   "1726376971",
    "@new_york_city_usa_wip":             "696414736",
    "@northern_fronts_scandinaviaww2":      "1507586063",
    "@northern_fronts_ace_compat":             "1507963938",
    "@project_h2h":           				  "2150435686",
    "@project_injury_reaction":      	       "1665585720",
    "@pulau":           					  "1423583812",
    "@reshmaan_province":     		        "843362862",
    "@rhsafrf":        					     "843425103",
    "@rhsgref":         				    "843593391",
    "@rhspkl":          				   "1978754337",
    "@rhssaf":         					    "843632231",
    "@rhsusaf":        					     "843577117",
    "@rkl_studios_attachments_v302":             "1661066023",
    "@stalker_live_zone":           			  "1975948563",
    "@santa_claus":          				   "567871211",
	"@secret_weapons":        				     "756352410",
	"@service_and_supply":             "2183975396",
    "@task_force_radio":                                     "620019431",
    "@tembelan_island":         			    "1252091296",
	"@the_pacific_war_gilbert_and_marshall":             "2232098886",
    "@the_pacific_war_solomon_islands":             "2203819698",
    "@tier_one_weapons":            			 "2268351256",
    "@usaf_mod_fighters":         				    "1891343103",
    "@usaf_mod_main_pack":        			     "1891346714",
    "@usaf_mod_utility":         			    "1891349162",
    "@usaf_mod_ac130":         				    "2226368165",
    "@veteran_mod":          				   "1774055232",
    "@vindicta":           						  "2185874952",
	"@wmo_walkable_moving_objects":             "925018569",
    "@zeus_enhanced":            			 "1779063631",
	"@zeus_enhanced_ace3_compat":             "2018593688",
    "@zombies_and_demons":       		      "501966277"

}

PATTERN = re.compile(r"workshopAnnouncement.*?<p id=\"(\d+)\">", re.DOTALL)
WORKSHOP_CHANGELOG_URL = "https://steamcommunity.com/sharedfiles/filedetails/changelog"
#endregion

#region Functions
def log(msg):
    print("")
    print("{{0:=<{}}}".format(len(msg)).format(""))
    print(msg);
    print("{{0:=<{}}}".format(len(msg)).format(""))


def call_steamcmd(params):
    os.system("{} {}".format(STEAM_CMD, params))
    print("")


def update_server():
    steam_cmd_params  = " +login {} {}".format(STEAM_USER, STEAM_PASS)
    steam_cmd_params += " +force_install_dir /home/arma3/arma3/"
    steam_cmd_params += " +app_update {} -beta creatordlc validate".format(A3_SERVER_ID)
    steam_cmd_params += " +quit"

    call_steamcmd(steam_cmd_params)


def mod_needs_update(mod_id, path):
    if os.path.isdir(path):
        response = request.urlopen("{}/{}".format(WORKSHOP_CHANGELOG_URL, mod_id)).read()
        response = response.decode("utf-8")
        match = PATTERN.search(response)

        if match:
            updated_at = datetime.fromtimestamp(int(match.group(1)))
            created_at = datetime.fromtimestamp(os.path.getctime(path))

            return (updated_at >= created_at)

    return False


def update_mods():
    for mod_name, mod_id in MODS.items():
        path = "/home/arma3/Steam/steamapps/workshop/content/107410/{}".format(mod_id)

        # Check if mod needs to be updated
        if os.path.isdir(path):

            if mod_needs_update(mod_id, path):
                # Delete existing folder so that we can verify whether the
                # download succeeded
                shutil.rmtree(path)
            else:
                print("No update required for \"{}\" ({})... SKIPPING".format(mod_name, mod_id))
                continue

        # Keep trying until the download actually succeeded
        tries = 0
        while os.path.isdir(path) == False and tries < 10:
            log("Updating \"{}\" ({}) | {}".format(mod_name, mod_id, tries + 1))

            steam_cmd_params  = " +login {} {}".format(STEAM_USER, STEAM_PASS)
            steam_cmd_params += " +force_install_dir /home/arma3/Steam/"
            steam_cmd_params += " +workshop_download_item {} {} validate".format(
                A3_WORKSHOP_ID,
                mod_id
            )
            steam_cmd_params += " +quit"

            call_steamcmd(steam_cmd_params)

            # Sleep for a bit so that we can kill the script if needed
            time.sleep(5)

            tries = tries + 1

        if tries >= 10:
            log("!! Updating {} failed after {} tries !!".format(mod_name, tries))


def lowercase_workshop_dir():
    os.system("(cd /home/arma3/Steam/steamapps/workshop/content/107410/ && ./majuscules.sh)")


def create_mod_symlinks():
    for mod_name, mod_id in MODS.items():
        link_path = "/home/arma3/arma3/{}".format(mod_name)
        real_path = "/home/arma3/Steam/steamapps/workshop/content/107410/{}".format(mod_id)

        if os.path.isdir(real_path):
            if not os.path.islink(link_path):
                os.symlink(real_path, link_path)
                print("Creating symlink '{}'...".format(link_path))
        else:
            print("Mod '{}' does not exist! ({})".format(mod_name, real_path))
#endregion

log("Updating A3 server ({})".format(A3_SERVER_ID))
update_server()

log("Updating mods")
update_mods()

log("Converting uppercase files/folders to lowercase...")
lowercase_workshop_dir()

log("Creating symlinks...")
create_mod_symlinks()

Then, i've added this other script here: /home/arma3/Steam/steamapps/workshop/content/107410/majuscules.sh

#!/bin/bash
# Rename all directories. This will need to be done first.
# Process each directory^rs contents before the directory itself
find * -depth -type d | while read x
do
        # Translate Caps to Small letters
        y=$(echo "$x" | tr '[A-Z ]' '[a-z_]');

        # create directory if it does not exit
        if [ ! -d "$y" ]; then
                mkdir -p "$y";
        fi

        # check if the source and destination is the same
        if [ "$x" != "$y" ]; then

                # move directory files before deleting
                ls -A "$x" | while read i
                do
                  mv "$x"/"$i" "$y";
                done
                rmdir "$x";

        fi

done

# Rename all files
find * -type f | while read x ;
do
        # Translate Caps to Small letters
        y=$(echo "$x" | tr '[A-Z ]' '[a-z_]');
        if [ "$x" != "$y" ]; then
                mv "$x" "$y";
        fi
done

exit 0

Every time I run "python3 firstscript.py" arma server and all mods are checked and updated (if needed), all files and folders turned into lowercase and symlinks added to the game directory. It's not as easy as adding the mods via the panel but it gets de job done.