Add more entries to robots.txt (#2044)

* Add more entries to `robots.txt`

* remove some entries as discussed

---------

Co-authored-by: Masterjun3 <[email protected]>

Author: YoshiRulz

TASVideos/Middleware/RobotHandlingMiddleware.cs

@@ -17,8 +17,12 @@ public async Task Invoke(HttpContext context)
 						Disallow: /MovieMaintenanceLog
 						Disallow: /UserMaintenanceLog
 						Disallow: /InternalSystem/
+						Disallow: /Search
 						Disallow: /*?revision=*
 						Disallow: /Wiki/PageHistory
+						Disallow: /Wiki/PageNotFound
+						Disallow: /Wiki/Referrers
+						Disallow: /Wiki/ViewSource
 
 						User-agent: Fasterfox
 						Disallow: /