azure


Programmatically Get VM Instance Network & Memory Info


This question is for Continuous Web Jobs.
Main Questions
How can we "VIEW" or programmatically "LOG" the current memory & network status of a VM running a Continuous Web Job?
Background:
Our web job is scraping some API and we keep getting 500 errors. We believe that the VM is firing too many threads for API requests - and then because of network limitations - when the responses come back, too many responses come back at the same time, overloading the VM's network limitations.
Side Questions:
How would you use MS Azure to Web scrape - and make sure you don't overload (in terms of memory + network) the VM it's running on?
(It seems that for background processing, these VMs are built for CPU calculation - not for Web/API scraping)
I'm still using the Monitoring (Classic) APIs currently. I've not found a "non-classic" version of the API, but I've also not spent much time looking. Since a web job runs as part of the Web App you'll need to monitor the web app using the tools provided in the Microsoft.WindowsAzure.Management.Monitoring.Metrics Namespace.
I found the API to be somewhat confusing, but spent sometime working with the PG to get it right. I've provided some sample code on the MSPFE github page at: https://github.com/mspfe/AzureMetricsAPISampleKit. Running the "tests" in this Solution will show you how to use the lib.
You first need to identify the web app by getting a list of them:
var webSpaceList = _webSiteClient.WebSpaces.List();
Then collect the availabile metrics:
foreach (var website in websiteList)
{
MetricDefinitionListResponse wsMetricListResponse = _metricsClient.MetricDefinitions.List(website.WebsiteResourceId, null, null);
website.MetricDefinitionsList = wsMetricListResponse.MetricDefinitionCollection;
website.MetricNamesList = new List<string>();
foreach (var metric in website.MetricDefinitionsList.Value)
{
website.MetricNamesList.Add(metric.Name);
}
MetricValueListResponse wsValueResponse = _metricsClient.MetricValues.List(website.WebsiteResourceId, website.MetricNamesList, "",
_timeGrain, _startDateTime, _endDateTime);
website.MetricValueList = wsValueResponse.MetricValueSetCollection;
}
From there you should have metric definitions and values. Sorry if this code is a little dated... but it should work.
Azure WebJobs run within your Azure App Service's web app (formerly called Websites). So, your capacity is governed by the size (and quantity) of Web App instances, whether free tier or one of the paid tiers. And you'd measure your utilization against the Web App instances.
Your side question, about how to use Azure to web scrape, is not answerable here: It's an opinion-based question with no right answer.

Related Links

Copying file contents from Azure Storage to Azure SQL Db using Azure Data Factory
Azure App service API app host vs API app
Cron jobs in Azure: Automation, Scheduler, WebJobs or other solution?
Azure App Service - Issue with upload files via SSL
Bibucket failed to deploy to azure “Failed to acquire content from your repository”
How to pass Body Parameters to Post operation using API Management Action in Logic App?
Azure AD Web App with zombie Login Redirect Loop
External authentication not working when configured in 'Authentication / Authorization' settings of Azure portal
How to iterate through all files in sftp folder in Microsoft Azure Logic App
Set VSO build/release number as Azure application setting
How to increase size of OS disk in Windows Azure using Powershell
Azure - Managed Disks, how to create snapshot
Azure App Service Authentication Multiple Servers / Regions
Is it possible to use Azure CLI to configure deployment to an app service from a Dropbox location?
Cannot export a BACPAC from SQL Azure — ExtAuth_Key10 is not supported in Microsoft Azure SQL Database v12
Azure Portal - AD Group Members only lists first 100 members

Categories

HOME
bootstrap-4
performance-testing
jsessionid
typeahead.js
asana
quickblox
fpdf
robot
sony
sweetalert
documentation
range
command-line-interface
postgresql-9.5
group-by
numericupdown
orchardcms-1.10
ssrs-tablix
raspberry-pi2
vast
ssrs-2008
gas
snort
libpcap
dotnetbar
jpos
cayley
ormlite-servicestack
missingmethodexception
hough-transform
google-oauth2
intersystems
esri
typemock-isolator
file-manager
inria-spoon
edit-distance
delete-row
macvim
atlassian-bitbucketserver
protobuf-3
wso2iot
compass-lucene
jpype
slt
database-cleaner
masm32
maven-versions-plugin
harvest-scm
email-notifications
masspay
texture2d
python-3.1
storage-duration
genetic
domo
fab
dnx
python-2.3
spymemcached
push-api
treegrid
wss4j
bonfire
gem
gcovr
modern-ui
cvs2git
sbrk
imaging
cpp-netlib
nhunspell
declspec
dfsort
scala-pickling
re2c
mainwindow
impdp
pixelsense
server.mappath
objcopy
katta
gemstone
ramaze
rails-3.1
xoom
sqlitemanager
sqlprofileprovider
greensoftware

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App