Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenRecords v3.5 - Main #558

Open
wants to merge 55 commits into
base: main
Choose a base branch
from
Open
Changes from 5 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
068cfa1
Re-implement server-side session and session timeout
zgary Nov 10, 2020
453626b
Add Clear-Site-Data header on logout
zgary Nov 20, 2020
7ab2419
Install pyotp and qrious; add MFA table; WIP views
zgary Dec 7, 2020
9dbc957
Create MFA blueprint and implement MFA backend functionality
zgary Dec 23, 2020
f744108
Merge pull request #535 from CityOfNewYork/g-zhou-openrecords-session…
joelbcastillo Dec 29, 2020
6b7d054
Install pyotp and qrious; add MFA table; WIP views
zgary Dec 7, 2020
436a08c
Create MFA blueprint and implement MFA backend functionality
zgary Dec 23, 2020
130e019
Update MFA secret column to LargeBinary datatype
zgary Jan 4, 2021
2963535
Rebase fix
zgary Jan 4, 2021
6b3b4a6
Remove old migration
zgary Jan 4, 2021
3501805
Use file for fernet key and create utils file for fernet methods
zgary Jan 5, 2021
cf89430
Update python dependencies
zgary Jan 7, 2021
81a0698
Add button to MFA manage page and add language to MFA register and ve…
zgary Jan 8, 2021
f4f3c67
Update Pipfile
zgary Jan 8, 2021
f934244
Gunicorn in pipfile; update gunicorn_config.py
joelbcastillo Jan 11, 2021
213e569
Merge pull request #538 from CityOfNewYork/g-zhou-openrecords-mfa
joelbcastillo Jan 13, 2021
ac7b286
Fix bugs in mfa register and remove
zgary Jan 15, 2021
fa10535
Update celery_config and .env.example
zgary Jan 15, 2021
2a5add9
Changes per PR comments
zgary Jan 22, 2021
a8ce394
Properly set default value for CELERY_CLEAR_EXPIRED_SESSION_IDS_INTERVAL
zgary Jan 22, 2021
b790ad9
Merge pull request #539 from CityOfNewYork/g-zhou-fix-mfa-bugs
zgary Jan 25, 2021
476a236
Updated specific agency instructions for DOT
johnyu95 Oct 7, 2021
26816f8
Update python pip packages to latest versions
johnyu95 May 2, 2022
ba036b1
Fixed bugs from deprecated functions and code clean up
johnyu95 May 4, 2022
b564dc5
Fixed custom request forms bug for radios with no selected value
johnyu95 May 4, 2022
305c3b6
Updated Pipfile
johnyu95 May 12, 2022
26d9fc7
Merge pull request #553 from CityOfNewYork/johnyu95-update-packages
johnyu95 May 13, 2022
d7796e1
Added support for Azure storage on file uploads WIP
johnyu95 Aug 9, 2022
65cee08
Fixed file response uploads
johnyu95 Aug 23, 2022
b89f119
File response edit and delete for azure storage
johnyu95 Sep 6, 2022
cfbb8ad
Changed serving files from SAS tokens instead of downloading
johnyu95 Oct 12, 2022
291a4ba
Changed USE_SFTP checks to USE_VOLUME_STORAGE
johnyu95 Oct 13, 2022
68b0a9e
Fixed bug with uploads not working on volume storage
johnyu95 Oct 14, 2022
9e3bea9
Cleanup
johnyu95 Oct 17, 2022
0ee7ebd
Updated .env.example for Azure storage
johnyu95 Oct 17, 2022
88af1e8
Fixed response token check for anon/public users
johnyu95 Oct 17, 2022
7b3d6d1
Merged with develop
johnyu95 Nov 2, 2022
b974005
Updated deprecated celery setting names
johnyu95 Nov 10, 2022
fa02d8e
Added SFTP check for file deletion
johnyu95 Nov 10, 2022
800a9cd
Merge pull request #557 from CityOfNewYork/johnyu95-azure-storage
johnyu95 Nov 10, 2022
81cc7f4
Merged with develop and updated dependencies
johnyu95 Nov 18, 2022
f8271a1
Updated send_file parameter name
johnyu95 Nov 22, 2022
f54420a
Added agency name and acronym to es_update
johnyu95 Nov 30, 2022
26b6cb4
Removed unused code and added custom form name fix
johnyu95 Dec 15, 2022
040f1ee
Added option to toggle MFA
johnyu95 Dec 21, 2022
99169d7
Add minor code review changes
zgary Jan 20, 2023
b8f97df
Merge pull request #559 from CityOfNewYork/main-merge
zgary Jan 20, 2023
e7a3395
Fixed open data compliance report missing referrer header error
johnyu95 Jan 27, 2023
f19865d
Fixed open data compliance report missing referrer header error
johnyu95 Jan 27, 2023
95b4929
Update _generate_signature method to use HMAC-SHA256
zgary Feb 15, 2023
a9daa6f
Updated google translate styles
johnyu95 Feb 22, 2023
78da8ff
Fixed super user toggle bug
johnyu95 Mar 2, 2023
7609acc
Merge branch 'develop' into main-merge
johnyu95 Apr 1, 2024
1eae68e
Merged with main
johnyu95 Apr 1, 2024
52f7b88
Fixed typo in models.py
johnyu95 Apr 1, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .python-version
Original file line number Diff line number Diff line change
@@ -1 +1 @@
3.8.3
3.10.2
7 changes: 3 additions & 4 deletions Pipfile
Original file line number Diff line number Diff line change
@@ -4,6 +4,7 @@ url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
setuptools = "57.5.0"
pipdeptree = "*"
decorator = "*"
Faker = "*"
@@ -28,7 +29,6 @@ py = "*"

[packages]
alembic = "*"
anyjson = "*"
appdirs = "*"
asn1crypto = "*"
bcrypt = "*"
@@ -44,7 +44,7 @@ cryptography = "*"
cssselect2 = "*"
defusedxml = "*"
dominate = "*"
elasticsearch = "*"
elasticsearch = {version = ">=7.0,<8.0"}
holidays = "*"
html5lib = "*"
idna = "*"
@@ -82,7 +82,6 @@ business_calendar = "*"
CairoSVG = "*"
Flask = "*"
Flask-Bootstrap = "*"
Flask-Elasticsearch = "*"
Flask-Login = "*"
Flask-Mail = "*"
Flask-Migrate = "*"
@@ -117,4 +116,4 @@ python-magic = "*"
gunicorn = "*"

[requires]
python_version = "3.8.3"
python_version = "3.10.2"
1,833 changes: 1,155 additions & 678 deletions Pipfile.lock

Large diffs are not rendered by default.

12 changes: 5 additions & 7 deletions app/__init__.py
Original file line number Diff line number Diff line change
@@ -11,25 +11,25 @@
from celery import Celery
from flask import (Flask, abort, redirect, render_template, request as flask_request, session, url_for)
from flask_bootstrap import Bootstrap
from flask_elasticsearch import FlaskElasticsearch
from flask_login import LoginManager, current_user
from flask_mail import Mail
from flask_moment import Moment
from flask_sqlalchemy import SQLAlchemy
from flask_tracy import Tracy
from flask_wtf import CsrfProtect
from flask_wtf.csrf import CSRFProtect
from flask_session import Session
from raven.contrib.flask import Sentry

from app import celery_config
from app.constants import OPENRECORDS_DL_EMAIL
from app.lib import NYCHolidays, jinja_filters
from config import Config, config
from elasticsearch import Elasticsearch

bootstrap = Bootstrap()
es = FlaskElasticsearch()
es = Elasticsearch(Config.ELASTICSEARCH_HOST)
db = SQLAlchemy()
csrf = CsrfProtect()
csrf = CSRFProtect()
moment = Moment()
mail = Mail()
tracy = Tracy()
@@ -101,9 +101,6 @@ def create_app(config_name='default'):
app.jinja_env.filters['format_ultimate_determination_reason'] = jinja_filters.format_ultimate_determination_reason

bootstrap.init_app(app)
es.init_app(app,
use_ssl=app.config['ELASTICSEARCH_USE_SSL'],
verify_certs=app.config['ELASTICSEARCH_VERIFY_CERTS'])
db.init_app(app)
csrf.init_app(app)
moment.init_app(app)
@@ -113,6 +110,7 @@ def create_app(config_name='default'):
celery.config_from_object(celery_config)
sentry.init_app(app, logging=app.config["USE_SENTRY"], level=logging.INFO)
sess.init_app(app)
app.elasticsearch = Elasticsearch(Config.ELASTICSEARCH_HOST)

with app.app_context():
from app.models import Anonymous
8 changes: 4 additions & 4 deletions app/admin/forms.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
from flask_login import current_user
from flask_wtf import Form
from flask_wtf import FlaskForm
from wtforms import (
SelectField,
StringField,
@@ -15,7 +15,7 @@
from app.models import Agencies


class ActivateAgencyUserForm(Form):
class ActivateAgencyUserForm(FlaskForm):
users = SelectField('Add Agency Users')

def __init__(self, agency_ein):
@@ -25,7 +25,7 @@ def __init__(self, agency_ein):
for u in Agencies.query.filter_by(ein=agency_ein).one().inactive_users]


class SelectAgencyForm(Form):
class SelectAgencyForm(FlaskForm):
agencies = SelectField('Current Agency')

def __init__(self, agency_ein=None):
@@ -72,7 +72,7 @@ def __init__(self, agency_ein=None):
# TODO: Add forms to modify agency_features (see models.py:183)


class AddAgencyUserForm(Form):
class AddAgencyUserForm(FlaskForm):
agency = SelectField('Agency', choices=None, validators=[DataRequired()])
first_name = StringField('First Name', validators=[Length(max=32), DataRequired()])
last_name = StringField('Last Name', validators=[Length(max=64), DataRequired()])
2 changes: 2 additions & 0 deletions app/agency/api/views.py
Original file line number Diff line number Diff line change
@@ -91,6 +91,8 @@ def get_custom_request_form_options(agency_ein):
CustomRequestForms.category,
CustomRequestForms.minimum_required).filter_by(
agency_ein=agency_ein).order_by(asc(CustomRequestForms.category), asc(CustomRequestForms.id)).all()
# Convert the results of with_entities back to tuple format so that jsonify can be used
custom_request_forms = [tuple(form) for form in custom_request_forms]
return jsonify(custom_request_forms), 200


7 changes: 3 additions & 4 deletions app/auth/forms.py
Original file line number Diff line number Diff line change
@@ -5,15 +5,14 @@
:synopsis: Defines the forms used to manage Authentication requests
"""

from flask_wtf import Form
from flask_wtf import FlaskForm
from wtforms import StringField, SelectField, PasswordField, SubmitField
from wtforms.validators import Length, Email, Optional, DataRequired

from app.constants import STATES
from app.models import Agencies


class StripFieldsForm(Form):
class StripFieldsForm(FlaskForm):
"""
Any field data that can be stripped, will be stripped.
http://stackoverflow.com/questions/26232165/automatically-strip-all-values-in-wtforms
@@ -174,7 +173,7 @@ def validate(self):
)


class BasicLoginForm(Form):
class BasicLoginForm(FlaskForm):
email = StringField("Email")
password = PasswordField("Password")

6 changes: 2 additions & 4 deletions app/models.py
Original file line number Diff line number Diff line change
@@ -527,6 +527,8 @@ def is_agency_active(self, ein=None):

def agencies_for_forms(self):
agencies = self.agencies.with_entities(Agencies.ein, Agencies._name).all()
# Convert the results of with_entities back to tuple format so that agencies can be processed
agencies = [tuple(agency) for agency in agencies]
agencies.insert(
0,
agencies.pop(
@@ -590,7 +592,6 @@ def es_update(self):
es,
actions,
index=current_app.config["ELASTICSEARCH_INDEX"],
doc_type="request",
chunk_size=current_app.config["ELASTICSEARCH_CHUNK_SIZE"],
)

@@ -968,7 +969,6 @@ def es_update(self):
if self.agency.is_active:
es.update(
index=current_app.config["ELASTICSEARCH_INDEX"],
doc_type="request",
id=self.id,
body={
"doc": {
@@ -1004,7 +1004,6 @@ def es_create(self):
if current_app.config["ELASTICSEARCH_ENABLED"]:
es.create(
index=current_app.config["ELASTICSEARCH_INDEX"],
doc_type="request",
id=self.id,
body={
"title": self.title,
@@ -1040,7 +1039,6 @@ def es_delete(self):
if current_app.config["ELASTICSEARCH_ENABLED"]:
es.delete(
index=current_app.config["ELASTICSEARCH_INDEX"],
doc_type="request",
id=self.id,
)

10 changes: 5 additions & 5 deletions app/report/forms.py
Original file line number Diff line number Diff line change
@@ -6,15 +6,15 @@
from datetime import date, timedelta

from flask_login import current_user
from flask_wtf import Form
from flask_wtf import FlaskForm
from wtforms import DateField, SelectField, SubmitField
from wtforms.validators import DataRequired

from app.lib.db_utils import get_agency_choices
from app.constants.dates import MONTHS, PORTAL_START_YEAR


class ReportFilterForm(Form):
class ReportFilterForm(FlaskForm):
"""
Form for users to filter different report statistics.

@@ -37,7 +37,7 @@ def __init__(self):
self.agency.choices.insert(1, self.agency.choices.pop(self.agency.choices.index(user_agency)))


class AcknowledgmentForm(Form):
class AcknowledgmentForm(FlaskForm):
"""Form to generate a report with acknowledgment data."""
date_from = DateField('Date From (required)', format='%m/%d/%Y', validators=[DataRequired()])
date_to = DateField('Date To (required)', format='%m/%d/%Y', validators=[DataRequired()])
@@ -60,7 +60,7 @@ def validate(self):
return is_valid


class MonthlyMetricsReportForm(Form):
class MonthlyMetricsReportForm(FlaskForm):
"""Form to generate a monthly metrics report."""
year = SelectField('Year (required)', choices=None, validators=[DataRequired()])
month = SelectField('Month (required)', choices=MONTHS, validators=[DataRequired()])
@@ -76,7 +76,7 @@ def __init__(self):
self.year.choices.insert(0, ('', ''))


class OpenDataReportForm(Form):
class OpenDataReportForm(FlaskForm):
"""Form to generate a report with Open Data compliance data."""
date_from = DateField('Date From (required)', id='open-data-date-from', format='%m/%d/%Y', validators=[DataRequired()])
date_to = DateField('Date To (required)', id='open-data-date-to', format='%m/%d/%Y', validators=[DataRequired()])
22 changes: 11 additions & 11 deletions app/request/forms.py
Original file line number Diff line number Diff line change
@@ -7,7 +7,7 @@
from datetime import datetime

from flask_login import current_user
from flask_wtf import Form
from flask_wtf import FlaskForm
from flask_wtf.file import FileField
from wtforms import (
StringField,
@@ -32,7 +32,7 @@
from app.lib.recaptcha_utils import Recaptcha3Field


class PublicUserRequestForm(Form):
class PublicUserRequestForm(FlaskForm):
"""
Form for public users to create a new FOIL request.
For a public user, the required fields are:
@@ -65,7 +65,7 @@ def __init__(self):
self.request_agency.choices.insert(0, ("", ""))


class AgencyUserRequestForm(Form):
class AgencyUserRequestForm(FlaskForm):
"""
Form for agency users to create a new FOIL request.
For an agency user, the required fields are:
@@ -133,7 +133,7 @@ def __init__(self):
self.request_agency.choices = current_user.agencies_for_forms()


class AnonymousRequestForm(Form):
class AnonymousRequestForm(FlaskForm):
"""
Form for anonymous users to create a new FOIL request.
For a anonymous user, the required fields are:
@@ -189,7 +189,7 @@ def __init__(self):
self.request_agency.choices.insert(0, ("", ""))


class EditRequesterForm(Form):
class EditRequesterForm(FlaskForm):
# TODO: Add class docstring
email = StringField("Email")
phone = StringField("Phone Number")
@@ -220,7 +220,7 @@ def __init__(self, requester):
self.zipcode.data = requester.mailing_address.get("zip") or ""


class DeterminationForm(Form):
class DeterminationForm(FlaskForm):
# TODO: Add class docstring
def __init__(self, agency_ein):
super(DeterminationForm, self).__init__()
@@ -309,7 +309,7 @@ class ReopenRequestForm(DeterminationForm):
ultimate_determination_type = [determination_type.REOPENING]


class GenerateEnvelopeForm(Form):
class GenerateEnvelopeForm(FlaskForm):
# TODO: Add class docstring
template = SelectField("Template")
recipient_name = StringField("Recipient Name")
@@ -341,7 +341,7 @@ def __init__(self, agency_ein, requester):
self.zipcode.data = requester.mailing_address.get("zip") or ""


class GenerateLetterForm(Form):
class GenerateLetterForm(FlaskForm):
# TODO: Add class docstring
def __init__(self, agency_ein):
super(GenerateLetterForm, self).__init__()
@@ -439,7 +439,7 @@ class GenerateResponseLetterForm(GenerateLetterForm):
letter_type = [response_type.LETTER]


class SearchRequestsForm(Form):
class SearchRequestsForm(FlaskForm):
# TODO: Add class docstring
agency_ein = SelectField("Agency")
agency_user = SelectField("User")
@@ -507,7 +507,7 @@ def __init__(self):
self.process()


class ContactAgencyForm(Form):
class ContactAgencyForm(FlaskForm):
# TODO: Add class docstring
first_name = StringField(
u"First Name", validators=[InputRequired(), Length(max=32)]
@@ -531,5 +531,5 @@ def __init__(self, request):
self.subject.data = "Inquiry about {}".format(request.id)


class TechnicalSupportForm(Form):
class TechnicalSupportForm(FlaskForm):
recaptcha = Recaptcha3Field(action="TestAction", execute_on_load=True)
4 changes: 2 additions & 2 deletions app/request/views.py
Original file line number Diff line number Diff line change
@@ -20,7 +20,7 @@
from flask_login import current_user
from sqlalchemy import any_
from sqlalchemy.orm.exc import NoResultFound
from werkzeug.utils import escape
from markupsafe import escape

from app.constants import request_status, permission, HIDDEN_AGENCIES
from app.lib.date_utils import DEFAULT_YEARS_HOLIDAY_LIST, get_holidays_date_list
@@ -79,7 +79,7 @@ def new():
kiosk_mode = eval_request_bool(escape(flask_request.args.get("kiosk_mode", False)))
category = str(escape(flask_request.args.get("category", None)))
agency = str(escape(flask_request.args.get("agency", None)))
title = str(escape(flask_request.args.get("title", None)))
title = str(escape(flask_request.args.get("title", "")))

if current_user.is_public:
form = PublicUserRequestForm()
Loading