code/Lab4_UI_part2.ipynb

{"cells":[{"cell_type":"markdown","source":["** Explore Bus Trips! **\n\nThis app allow you to do Data exploration.\n\nAll you need to do is specifiy the data path (or IP if streaming), and start to explore by using our interative dashboard, automatic visualizaion platform or quick sampled data viz!\n\n*More detailed explanation below."],"metadata":{}},{"cell_type":"code","source":["\nfrom IPython.display import Image\nImage(url=\"https://drive.google.com/uc?id=1S13K-zCiK2nLgY9pdSSjgntxyOl-nm-x\",  width=200)"],"metadata":{},"outputs":[{"output_type":"display_data","metadata":{},"output_type":"display_data","data":{"text/html":["<img src=\"https://drive.google.com/uc?id=1S13K-zCiK2nLgY9pdSSjgntxyOl-nm-x\" width=\"200\"/>"]}}],"execution_count":2},{"cell_type":"markdown","source":["**Dashboard**\n\nTo filter the Interactive Dashboard use the different filters below, or directly select \\ click on the part you are interested in. \n\nAlternatively, use the query bar below (for example: \"vehicleId :33317 or vehicleId :33318\" to see only these vehicles)."],"metadata":{}},{"cell_type":"code","source":["!pip install libify\nimport libify\nfrom pyspark.sql.functions import date_format\nimport pyspark.sql.functions as F\n\nlab4_functions = libify.importer(globals(), '/Users/shubi@campus.technion.ac.il/Lab4_functions')\n\ndbutils.widgets.dropdown(\"Stream?\", \"False\", ['True', 'False'])\ndbutils.widgets.text(\"IP / Path\", '')\n\nisStream = eval(dbutils.widgets.get(\"Stream?\"))\npath_ip = dbutils.widgets.get(\"IP / Path\")\n\n# Our check\n# 10.0.0.30:9091\n# /mnt/dacoursedatabricksstg/dacoursedatabricksdata/busFile \ndf = lab4_functions.create_warehouse_df(streaming=isStream, USE_TAKE=False, min_date=None, max_date=None, path_ip=path_ip)\nif df == 'problem':\n    print('\\n\\n\\n\\n')\n    print('Unexpected error in reading file.\\n'\n          'Please make sure the file exists, ' \n          'is of the correct format(json) and with the right schema.')\ndf = lab4_functions.hydrate_data(df)\ndf = lab4_functions.aggregate_data(df, streaming=isStream)\ndf = df.drop('congestionPercentage')\nif isStream:\n    df = df.withColumn('day', date_format(\"TimestampDay\", 'yyyy-MM-dd')).drop(\"TimestampDay\")\ndf = df.withColumn('isEventDate', F.col('isEventDate').cast('int'))\\\n       .withColumn('isPassedLocation', F.col('isPassedLocation').cast('int'))    \nlab4_functions.write_to_elastic(df, 'group-data-to-explore', lab4_functions.elastic_settings_mappings_date, overwrite=False, streaming=isStream)"],"metadata":{},"outputs":[{"output_type":"display_data","metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":4},{"cell_type":"code","source":["displayHTML(\"\"\"<iframe src=\"https://da2020w-0001.eastus.cloudapp.azure.com:5601/goto/9ce95d281a2ae62b5b92202efa43a7ca\" height=\"1013\" width=\"1800\"></iframe>\"\"\")"],"metadata":{},"outputs":[{"output_type":"display_data","metadata":{},"output_type":"display_data","data":{"text/html":["<iframe src=\"https://da2020w-0001.eastus.cloudapp.azure.com:5601/goto/9ce95d281a2ae62b5b92202efa43a7ca\" height=\"1013\" width=\"1800\"></iframe>"]}}],"execution_count":5},{"cell_type":"markdown","source":["**Automatic visualizaion platform** - *visualizations with a single mouse gesture* - Drag & Drop a field of interest into the center part of the display. \n\nThen, pick one of smart suggestions of ways to visualize your data.\n\nAlternatively, easily change the underlying query, visualization type or filters, all with an easy-to-use, intuitive UI!"],"metadata":{}},{"cell_type":"code","source":["displayHTML(\"\"\"<iframe src=\"https://da2020w-0001.eastus.cloudapp.azure.com:5601/app/lens#/?_g=(filters:!(),refreshInterval:(pause:!f,value:30000),time:(from:'2017-03-01T10:56:37.000Z',to:'2018-09-11T18:10:43.000Z'))\" height=\"1013\" width=\"1800\"></iframe>\"\"\")"],"metadata":{},"outputs":[{"output_type":"display_data","metadata":{},"output_type":"display_data","data":{"text/html":["<iframe src=\"https://da2020w-0001.eastus.cloudapp.azure.com:5601/app/lens#/?_g=(filters:!(),refreshInterval:(pause:!f,value:30000),time:(from:'2017-03-01T10:56:37.000Z',to:'2018-09-11T18:10:43.000Z'))\" height=\"1013\" width=\"1800\"></iframe>"]}}],"execution_count":7},{"cell_type":"markdown","source":["**Visualization using sampled data**\n\nThis interface allows to quickly get insights about the general distribution of the data, and some statistics about it.\nEven on large database, by selecting a sample size you can get fast results. \nAdditionally, you can query the database and get approximate results as well. \n\nUpdated in realtime."],"metadata":{}},{"cell_type":"code","source":["displayHTML(\"\"\"<iframe src=\"https://da2020w-0001.eastus.cloudapp.azure.com:5601/app/ml#/jobs/new_job/datavisualizer?_g=%28refreshInterval%3A%28pause%3A%21f%2Cvalue%3A30000%29%2Ctime%3A%28from%3A%272017-07-03T00%3A00%3A00.000Z%27%2Cto%3A%272018-02-13T00%3A00%3A00.000Z%27%29%29&index=b17555c0-574c-11eb-b032-278f6d866662\" height=\"3000\" width=\"1800\"></iframe>\"\"\")"],"metadata":{},"outputs":[{"output_type":"display_data","metadata":{},"output_type":"display_data","data":{"text/html":["<iframe src=\"https://da2020w-0001.eastus.cloudapp.azure.com:5601/app/ml#/jobs/new_job/datavisualizer?_g=%28refreshInterval%3A%28pause%3A%21f%2Cvalue%3A30000%29%2Ctime%3A%28from%3A%272017-07-03T00%3A00%3A00.000Z%27%2Cto%3A%272018-02-13T00%3A00%3A00.000Z%27%29%29&index=b17555c0-574c-11eb-b032-278f6d866662\" height=\"3000\" width=\"1800\"></iframe>"]}}],"execution_count":9},{"cell_type":"markdown","source":["* Choose the data to run on by adding ip number if it is a streaming data, and a file path if it is a batch data.\n* Run by pressing on the \"update\" button that appears at the top right of the app. (It might pop-up a massage, please press 'yes')\n* -> The visualizations automaticaly will be updated! Now you can explore in a convient and fast way.\n* Notes: \n  1. If the map is visualizations you might entered a wrong ip number.\n  2. The running might take a while.\n  3. If you see a pop-up massage \"Run beyond time out\" press it to continue.\n  4. if you are getting error \"Failed to construct kafka consumer\" - you might entered a wrong ip number."],"metadata":{}}],"metadata":{"name":"Lab4_UI_part2","notebookId":2483473424244723},"nbformat":4,"nbformat_minor":0}