azure


Programmatically Get VM Instance Network & Memory Info


This question is for Continuous Web Jobs.
Main Questions
How can we "VIEW" or programmatically "LOG" the current memory & network status of a VM running a Continuous Web Job?
Background:
Our web job is scraping some API and we keep getting 500 errors. We believe that the VM is firing too many threads for API requests - and then because of network limitations - when the responses come back, too many responses come back at the same time, overloading the VM's network limitations.
Side Questions:
How would you use MS Azure to Web scrape - and make sure you don't overload (in terms of memory + network) the VM it's running on?
(It seems that for background processing, these VMs are built for CPU calculation - not for Web/API scraping)
I'm still using the Monitoring (Classic) APIs currently. I've not found a "non-classic" version of the API, but I've also not spent much time looking. Since a web job runs as part of the Web App you'll need to monitor the web app using the tools provided in the Microsoft.WindowsAzure.Management.Monitoring.Metrics Namespace.
I found the API to be somewhat confusing, but spent sometime working with the PG to get it right. I've provided some sample code on the MSPFE github page at: https://github.com/mspfe/AzureMetricsAPISampleKit. Running the "tests" in this Solution will show you how to use the lib.
You first need to identify the web app by getting a list of them:
var webSpaceList = _webSiteClient.WebSpaces.List();
Then collect the availabile metrics:
foreach (var website in websiteList)
{
MetricDefinitionListResponse wsMetricListResponse = _metricsClient.MetricDefinitions.List(website.WebsiteResourceId, null, null);
website.MetricDefinitionsList = wsMetricListResponse.MetricDefinitionCollection;
website.MetricNamesList = new List<string>();
foreach (var metric in website.MetricDefinitionsList.Value)
{
website.MetricNamesList.Add(metric.Name);
}
MetricValueListResponse wsValueResponse = _metricsClient.MetricValues.List(website.WebsiteResourceId, website.MetricNamesList, "",
_timeGrain, _startDateTime, _endDateTime);
website.MetricValueList = wsValueResponse.MetricValueSetCollection;
}
From there you should have metric definitions and values. Sorry if this code is a little dated... but it should work.
Azure WebJobs run within your Azure App Service's web app (formerly called Websites). So, your capacity is governed by the size (and quantity) of Web App instances, whether free tier or one of the paid tiers. And you'd measure your utilization against the Web App instances.
Your side question, about how to use Azure to web scrape, is not answerable here: It's an opinion-based question with no right answer.

Related Links

DNS: Azure Server <-> SRV record (Godaddy)<-> Windows 2012 Domain Controller
Azure scalable architecture design
Looking for clarity on Event Hubs vs Topics in Azure Service Bus
How to provision multiple Azure VMs with DSC
Creating Azure Search
Set the max size of Particular.ServiceControl queue in azure
Update app.config on Azure worker role start up
can nlog configuration be set within azure worker role cloudconfiguration files
App.config transformations for multiple worker roles in Azure
Is it possible for the same worker to listen to 2 different subscriptions for different topics
Authenticating machine using Azure AD
Is there a way to specify a machine key for Azure Websites without specifying it in the web.config?
Windows Puppet Agent does not connect to Puppet Enterprise Master
400 Bad Request with Guzzle from the Microsoft Oauth endpoint
Remote monitoring Azure Products
MS Azure - automatically download the latest ip ranges used by MS Azure Datacenters

Categories

HOME
ios
matlab
clojure
logstash
image-segmentation
arm
open-source
visual-studio-2012
splunk
asp.net-mvc-3
wav
google-maps-sdk-ios
pymc3
coreos
message
export-to-excel
smartsheet-c#-sdk-v2
prerender
org-mode
zap
hp-alm
collision
man
code-analysis
datamatrix
py4j
jqplot
naturallyspeaking
structure
toolbar
ose
mobilefirst-bluemix
chamilo-lms
draggable
weebly
tibco-ems
protobuf-3
ibm-wcm
psychopy
polymer-cli
ulimit
keyboard-maestro
appdynamics
uservoice
multistore
jquery-masonry
xssf
okular
infiniband
omniauth
swift-package-manager
singlepage
breach-attack
shift-reduce-conflict
informatica-cloud
topojson
cakephp-2.3
ispconfig
vb4android
dms
oracle-policy-automation
blpapi
highslide
strace
metalkit
ssha
session-replication
heroku-toolbelt
cbc-mode
generalization
theorem-proving
sigkill
lossless-compression
image-scaling
nine-patch
wmv
sirtrevor
requiredfieldvalidator
declspec
delphi-2009
prudentia
geometry-surface
adlds
arbtt
icon-fonts
wescheme
applaud
rc-shell
code-structure
vim-fugitive
time-limiting
nsinvocation
expressionvisitor
httpcontext.cache
data-oriented-design
xmltextwriter
treetop
wiki-markup
spquery
xquery-update
polyglot
coords

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App