base on Hestia scrapes real estate websites for new rental listings ## Hestia Hestia scrapes real estate websites for new rental listings, and broadcasts the results via Telegram. Check out @hestia_homes_bot on Telegram: https://t.me/hestia_homes_bot ### How to contribute First of all, thanks! If you want to add a website, you need to write a parser. This takes a bit of detective work to find out how the website can be processed best. Ideally, if you check the requests from your browsers' inspector window, you see it makes a request to an API endpoint that gives you a clean JSON response with all the data you need. If that's the case, you can have a parser as simple as the REBO parser: ```python def parse_rebo(self, r: requests.models.Response): results = json.loads(r.content)["hits"] for res in results: home = Home(agency="rebo") home.address = res["address"] home.city = res["city"] home.url = "https://www.rebogroep.nl/nl/aanbod/" + res["slug"] home.price = int(res["price"]) self.homes.append(home) ``` Unfortunately, a lot of websites need parsing of the HTML body in order to get all the info. You can do this with BeautifulSoup, but usually requires some extra parsing like removing spaces and processing a price written as `€900,-` to get an integer. See the VBO parser for example: ```python def parse_vbo(self, r: requests.models.Response): results = BeautifulSoup(r.content, "html.parser").find_all("a", class_="propertyLink") for res in results: home = Home(agency="vbo") home.url = res["href"] home.address = res.select_one(".street").text.strip() home.city = res.select_one(".city").text.strip() rawprice = res.select_one(".price").text end = rawprice.index(",") # Every price is terminated with a trailing , home.price = int(rawprice[2:end].replace(".", "")) self.homes.append(home) ``` Some websites list homes that have already been rented out. You can filter them out in this code as well. Check the file `hestia.py`, this contains all the parsers for Hestia. The generic structure is that it takes an unprocessed `Response` object from the Python library `requests` (e.g. from `requests.get(url)`) and fills a `Home` object. If you wrote a parser and want to submit a PR, please include the following info: ``` Target URL GET or POST Headers (optional, in Python dict format) Request data (only for POST, in Python dict format) ``` I'll take your parser and run it in the dev environment for a few days to see how it performs. Does it process all homes correctly? Does the website modify their HTML structure every other day? Does the API endpoint need an updated ID every two weeks (looking at you, Woningnet...)? ### Additional contributors (thanks!): * [BLOKKADE](https://github.com/BLOKKADE) - NMG Wonen parser * [Rafaeltheraven](https://github.com/Rafaeltheraven) - VBO and Woonzeker parsers * [OmriSteiner](https://github.com/OmriSteiner) - Ooms and Atta parsers * [Ventilaar](https://github.com/ventilaar) - Hexia, Woonnet Rijnmond and Woonin parsers", Assign "at most 3 tags" to the expected json: {"id":"5056","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"